From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBBFF320A; Tue, 14 Mar 2023 11:03:00 +0000 (UTC) Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4PbVw92GJMz6J7Km; Tue, 14 Mar 2023 19:02:01 +0800 (CST) Received: from localhost (10.48.148.120) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Tue, 14 Mar 2023 11:02:51 +0000 Date: Tue, 14 Mar 2023 11:02:50 +0000 From: Jonathan Cameron To: James Morse CC: , , , , , , , , , , Marc Zyngier , Thomas Gleixner , Lorenzo Pieralisi , Mark Rutland , Sudeep Holla , Borislav Petkov , H Peter Anvin , Dave Hansen , Ingo Molnar , Will Deacon , Catalin Marinas , Huacai Chen , "Suzuki K Poulose" , Oliver Upton , Len Brown , Rafael Wysocki , "WANG Xuerui" , Salil Mehta , "Russell King" , Subject: Re: [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug Message-ID: <20230314110250.00005685@Huawei.com> In-Reply-To: <1f21673e-e5e6-a158-94a4-6ae6724c1f93@arm.com> References: <20230203135043.409192-1-james.morse@arm.com> <20230307120050.000032f1@Huawei.com> <1f21673e-e5e6-a158-94a4-6ae6724c1f93@arm.com> Followup-To: Jean-Philippe Brucker Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.48.148.120] X-ClientProxiedBy: lhrpeml500003.china.huawei.com (7.191.162.67) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected On Mon, 13 Mar 2023 15:50:52 +0000 James Morse wrote: > Hi Jonathan, >=20 > On 07/03/2023 12:00, Jonathan Cameron wrote: > > On Fri, 3 Feb 2023 13:50:11 +0000 > > James Morse wrote: =20 >=20 > >> On a system that supports cpuhotplug the MADT has to describe every po= ssible > >> CPU at boot. Under KVM, the vGIC needs to know about every possible vC= PU before > >> the guest is started. > >> With these constraints, virtual-cpuhotplug is really just a hypervisor= /firmware > >> policy about which CPUs can be brought online. > >> > >> This series adds support for virtual-cpuhotplug as exactly that: firmw= are > >> policy. This may even work on a physical machine too; for a guest the = part of > >> firmware is played by the VMM. (typically Qemu). > >> > >> PSCI support is modified to return 'DENIED' if the CPU can't be brought > >> online/enabled yet. The CPU object's _STA method's enabled bit is used= to > >> indicate firmware's current disposition. If the CPU has its enabled bi= t clear, > >> it will not be registered with sysfs, and attempts to bring it online = will > >> fail. The notifications that _STA has changed its value then work in t= he same > >> way as physical hotplug, and firmware can cause the CPU to be register= ed some > >> time later, allowing it to be brought online. =20 >=20 > > As we discussed on an LOD call a while back, I think that we need some = path to > > find out if the guest supports vCPU HP or not so that info can be queri= ed by > > an orchestrator / libvirt etc. In general the entity responsible for a= llocating > > extra vCPUs may not know what support the VM has for this feature. =20 >=20 > I agree. For arm64 this is going to be important if/when there are machin= es that do > physical hotplug of CPUs too. >=20 >=20 > > There are various ways we could get this information into the VMM. > > My immediate thought is to use one of the ACPI interfaces that lets us = write > > AML that can set an emulated register. A query to the VMM can check if = this > > register is set. > >=20 > > So options. > >=20 > > _OSI() - Deprecated on ARM64 so lets not use that ;) =20 >=20 > News to me, I've only just discovered it! >=20 >=20 > > _OSC() - Could add a bit to Table 6.13 Platform-Wide Capabilites in ACP= I 6.5 spec. > > Given x86 has a similar online capable bit perhaps this is the= best option > > though it is the one that requires a formal code first proposa= l to ASWG. =20 >=20 > I've had a go at writing this one: > https://gitlab.arm.com/linux-arm/linux-jm/-/commit/220b0d8b0261d7467c8705= e6f614d57325798859 =46rom a quick glance that looks good to me. >=20 > It'll appear in the v1 of the series once the kernel and qemu bits are al= l lined up again. We'll also need to kick off the spec change with a code-first proposal. I think current standard way to do that is a bugzilla entry in EDK2 repo=20 https://bugzilla.tianocore.org/buglist.cgi?component=3DSpecification%20Upda= te&product=3DEDK2%20Code%20First&resolution=3D--- and the get someone in ASWG to create equivalent tracking issue in mantis. Great if you already have that in hand via relevant ARM folks. Jonathan >=20 >=20 > Thanks, >=20 > James >=20 >=20 > > _OSC() - Could add a new UUID and put it under a suitable device - mayb= e all CPUs? > > You could definitely argue this feature is an operating system= property. > > _DSM() - Similar to OSC but always under a device. > > Whilst can be used for this I'm not sure it really matches int= ended usecase. > >=20 > > Assuming everyone agrees this bit of introspection is useful, > > Rafael / other ACPI specialists: Any suggestions on how best to do this= ? =20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jonathan Cameron Date: Tue, 14 Mar 2023 11:02:50 +0000 Subject: Re: [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug Message-Id: <20230314110250.00005685@Huawei.com> List-Id: References: <20230203135043.409192-1-james.morse@arm.com> <20230307120050.000032f1@Huawei.com> <1f21673e-e5e6-a158-94a4-6ae6724c1f93@arm.com> In-Reply-To: <1f21673e-e5e6-a158-94a4-6ae6724c1f93@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: James Morse Cc: linux-pm@vger.kernel.org, loongarch@lists.linux.dev, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org, linux-ia64@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, Marc Zyngier , Thomas Gleixner , Lorenzo Pieralisi , Mark Rutland , Sudeep Holla , Borislav Petkov , H Peter Anvin , Dave Hansen , Ingo Molnar , Will Deacon , Catalin Marinas , Huacai Chen , Suzuki K Poulose , Oliver Upton , Len Brown , Rafael Wysocki , WANG Xuerui , Salil Mehta , Russell King , kangkang.shen@futurewei.com On Mon, 13 Mar 2023 15:50:52 +0000 James Morse wrote: > Hi Jonathan, >=20 > On 07/03/2023 12:00, Jonathan Cameron wrote: > > On Fri, 3 Feb 2023 13:50:11 +0000 > > James Morse wrote: =20 >=20 > >> On a system that supports cpuhotplug the MADT has to describe every po= ssible > >> CPU at boot. Under KVM, the vGIC needs to know about every possible vC= PU before > >> the guest is started. > >> With these constraints, virtual-cpuhotplug is really just a hypervisor= /firmware > >> policy about which CPUs can be brought online. > >> > >> This series adds support for virtual-cpuhotplug as exactly that: firmw= are > >> policy. This may even work on a physical machine too; for a guest the = part of > >> firmware is played by the VMM. (typically Qemu). > >> > >> PSCI support is modified to return 'DENIED' if the CPU can't be brought > >> online/enabled yet. The CPU object's _STA method's enabled bit is used= to > >> indicate firmware's current disposition. If the CPU has its enabled bi= t clear, > >> it will not be registered with sysfs, and attempts to bring it online = will > >> fail. The notifications that _STA has changed its value then work in t= he same > >> way as physical hotplug, and firmware can cause the CPU to be register= ed some > >> time later, allowing it to be brought online. =20 >=20 > > As we discussed on an LOD call a while back, I think that we need some = path to > > find out if the guest supports vCPU HP or not so that info can be queri= ed by > > an orchestrator / libvirt etc. In general the entity responsible for a= llocating > > extra vCPUs may not know what support the VM has for this feature. =20 >=20 > I agree. For arm64 this is going to be important if/when there are machin= es that do > physical hotplug of CPUs too. >=20 >=20 > > There are various ways we could get this information into the VMM. > > My immediate thought is to use one of the ACPI interfaces that lets us = write > > AML that can set an emulated register. A query to the VMM can check if = this > > register is set. > >=20 > > So options. > >=20 > > _OSI() - Deprecated on ARM64 so lets not use that ;) =20 >=20 > News to me, I've only just discovered it! >=20 >=20 > > _OSC() - Could add a bit to Table 6.13 Platform-Wide Capabilites in ACP= I 6.5 spec. > > Given x86 has a similar online capable bit perhaps this is the= best option > > though it is the one that requires a formal code first proposa= l to ASWG. =20 >=20 > I've had a go at writing this one: > https://gitlab.arm.com/linux-arm/linux-jm/-/commit/220b0d8b0261d7467c8705= e6f614d57325798859 >From a quick glance that looks good to me. >=20 > It'll appear in the v1 of the series once the kernel and qemu bits are al= l lined up again. We'll also need to kick off the spec change with a code-first proposal. I think current standard way to do that is a bugzilla entry in EDK2 repo=20 https://bugzilla.tianocore.org/buglist.cgi?component=3DSpecification%20Upda= te&product=EDK2%20Code%20First&resolution=3D--- and the get someone in ASWG to create equivalent tracking issue in mantis. Great if you already have that in hand via relevant ARM folks. Jonathan >=20 >=20 > Thanks, >=20 > James >=20 >=20 > > _OSC() - Could add a new UUID and put it under a suitable device - mayb= e all CPUs? > > You could definitely argue this feature is an operating system= property. > > _DSM() - Similar to OSC but always under a device. > > Whilst can be used for this I'm not sure it really matches int= ended usecase. > >=20 > > Assuming everyone agrees this bit of introspection is useful, > > Rafael / other ACPI specialists: Any suggestions on how best to do this= ? =20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0725AC7618E for ; Tue, 14 Mar 2023 11:04:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Subject:CC:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=g9eQ2YyciLfKTlPfICv6fup78XPcf5hrpbzXBU8UOxw=; b=RdkgMCbaceOrxs k8p9NSNmw763EnUkRjTmllYjDYa8RC6k1Ligc+cqEtHmKiCPcxPMIYKfWNAGuwVtSki3bBlWiFcz6 uT1yv87EEVB4qdssJ0ytNoD7Kn78lmD5XzVaRW1ewqj5NediC9D/io+HAmZz+9OJoZTc5cOijy8QV HDESpUTv4zjogmhmzU2Yx7YzhvTYH5oeAfjiQDztUzF+GR819ZFKs5ulsPn0Ene0+JhfbiWyzRslm 2LIamQcAvrW87Yqe0eJO/i4xIq1SE5i7a86VkWqbOPci7sJaGT1toybW/OBAc5fZr5mMDrrR8Cru2 RDMxagRUsxySbW9ZT5hg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pc2Qt-009qMw-0R; Tue, 14 Mar 2023 11:03:09 +0000 Received: from frasgout.his.huawei.com ([185.176.79.56]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pc2Qp-009qLd-0S for linux-arm-kernel@lists.infradead.org; Tue, 14 Mar 2023 11:03:05 +0000 Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4PbVw92GJMz6J7Km; Tue, 14 Mar 2023 19:02:01 +0800 (CST) Received: from localhost (10.48.148.120) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Tue, 14 Mar 2023 11:02:51 +0000 Date: Tue, 14 Mar 2023 11:02:50 +0000 From: Jonathan Cameron To: James Morse CC: , , , , , , , , , , Marc Zyngier , Thomas Gleixner , Lorenzo Pieralisi , Mark Rutland , Sudeep Holla , Borislav Petkov , H Peter Anvin , Dave Hansen , Ingo Molnar , Will Deacon , Catalin Marinas , Huacai Chen , "Suzuki K Poulose" , Oliver Upton , Len Brown , Rafael Wysocki , "WANG Xuerui" , Salil Mehta , "Russell King" , Subject: Re: [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug Message-ID: <20230314110250.00005685@Huawei.com> In-Reply-To: <1f21673e-e5e6-a158-94a4-6ae6724c1f93@arm.com> References: <20230203135043.409192-1-james.morse@arm.com> <20230307120050.000032f1@Huawei.com> <1f21673e-e5e6-a158-94a4-6ae6724c1f93@arm.com> Followup-To: Jean-Philippe Brucker Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 X-Originating-IP: [10.48.148.120] X-ClientProxiedBy: lhrpeml500003.china.huawei.com (7.191.162.67) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230314_040303_482330_AB9E8C26 X-CRM114-Status: GOOD ( 38.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, 13 Mar 2023 15:50:52 +0000 James Morse wrote: > Hi Jonathan, > > On 07/03/2023 12:00, Jonathan Cameron wrote: > > On Fri, 3 Feb 2023 13:50:11 +0000 > > James Morse wrote: > > >> On a system that supports cpuhotplug the MADT has to describe every possible > >> CPU at boot. Under KVM, the vGIC needs to know about every possible vCPU before > >> the guest is started. > >> With these constraints, virtual-cpuhotplug is really just a hypervisor/firmware > >> policy about which CPUs can be brought online. > >> > >> This series adds support for virtual-cpuhotplug as exactly that: firmware > >> policy. This may even work on a physical machine too; for a guest the part of > >> firmware is played by the VMM. (typically Qemu). > >> > >> PSCI support is modified to return 'DENIED' if the CPU can't be brought > >> online/enabled yet. The CPU object's _STA method's enabled bit is used to > >> indicate firmware's current disposition. If the CPU has its enabled bit clear, > >> it will not be registered with sysfs, and attempts to bring it online will > >> fail. The notifications that _STA has changed its value then work in the same > >> way as physical hotplug, and firmware can cause the CPU to be registered some > >> time later, allowing it to be brought online. > > > As we discussed on an LOD call a while back, I think that we need some path to > > find out if the guest supports vCPU HP or not so that info can be queried by > > an orchestrator / libvirt etc. In general the entity responsible for allocating > > extra vCPUs may not know what support the VM has for this feature. > > I agree. For arm64 this is going to be important if/when there are machines that do > physical hotplug of CPUs too. > > > > There are various ways we could get this information into the VMM. > > My immediate thought is to use one of the ACPI interfaces that lets us write > > AML that can set an emulated register. A query to the VMM can check if this > > register is set. > > > > So options. > > > > _OSI() - Deprecated on ARM64 so lets not use that ;) > > News to me, I've only just discovered it! > > > > _OSC() - Could add a bit to Table 6.13 Platform-Wide Capabilites in ACPI 6.5 spec. > > Given x86 has a similar online capable bit perhaps this is the best option > > though it is the one that requires a formal code first proposal to ASWG. > > I've had a go at writing this one: > https://gitlab.arm.com/linux-arm/linux-jm/-/commit/220b0d8b0261d7467c8705e6f614d57325798859 >From a quick glance that looks good to me. > > It'll appear in the v1 of the series once the kernel and qemu bits are all lined up again. We'll also need to kick off the spec change with a code-first proposal. I think current standard way to do that is a bugzilla entry in EDK2 repo https://bugzilla.tianocore.org/buglist.cgi?component=Specification%20Update&product=EDK2%20Code%20First&resolution=--- and the get someone in ASWG to create equivalent tracking issue in mantis. Great if you already have that in hand via relevant ARM folks. Jonathan > > > Thanks, > > James > > > > _OSC() - Could add a new UUID and put it under a suitable device - maybe all CPUs? > > You could definitely argue this feature is an operating system property. > > _DSM() - Similar to OSC but always under a device. > > Whilst can be used for this I'm not sure it really matches intended usecase. > > > > Assuming everyone agrees this bit of introspection is useful, > > Rafael / other ACPI specialists: Any suggestions on how best to do this? > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel