From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out0.migadu.com (out0.migadu.com [94.23.1.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0974F3C38 for ; Fri, 20 Jan 2023 18:05:01 +0000 (UTC) Date: Fri, 20 Jan 2023 18:04:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1674237899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Z1CaMcH4x5rtypgK1Xxe771oK8a9hWqV3gyDwwh42eA=; b=s+eRmBypvtugBDgcUxeYCjsvJV1Ae4KYybdsroHaliFb55C2HFYchZaA8jsQLjwHC/lmtn kYYWKySUBrw07RcLprP4ZxCmfxU8Td+Cx8vDrKSZdQohAxRDZfWRZdgW+sPyJILUzSDNXb EnmsfZCqROpfaQcSS24cv2UsT7n+gCs= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Marc Zyngier Cc: Reiji Watanabe , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, James Morse , Alexandru Elisei , Zenghui Yu , Suzuki K Poulose , Paolo Bonzini , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata Subject: Re: [PATCH v2 3/8] KVM: arm64: PMU: Preserve vCPU's PMCR_EL0.N value on vCPU reset Message-ID: References: <20230117013542.371944-1-reijiw@google.com> <20230117013542.371944-4-reijiw@google.com> <86pmb9mmkv.wl-maz@kernel.org> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <86pmb9mmkv.wl-maz@kernel.org> X-Migadu-Flow: FLOW_OUT Hey Marc, On Fri, Jan 20, 2023 at 12:12:32PM +0000, Marc Zyngier wrote: > On Fri, 20 Jan 2023 00:30:33 +0000, Oliver Upton wrote: > > I think we need to derive a sanitised value for PMCR_EL0.N, as I believe > > nothing in the architecture prevents implementers from gluing together > > cores with varying numbers of PMCs. We probably haven't noticed it yet > > since it would appear all Arm designs have had 6 PMCs. > > This brings back the question of late onlining. How do you cope with > with the onlining of such a CPU that has a smaller set of counters > than its online counterparts? This is at odds with the way the PMU > code works. You're absolutely right, any illusion we derived from the online set of CPUs could fall apart with a late onlining of a different core. > If you have a different set of counters, you are likely to have a > different PMU altogether: > > [ 1.192606] hw perfevents: enabled with armv8_cortex_a57 PMU driver, 7 counters available > [ 1.201254] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available > > This isn't a broken system, but it has two set of cores which are > massively different, and two PMUs. > > This really should tie back to the PMU type we're counting on, and to > the set of CPUs that implements it. We already have some > infrastructure to check for the affinity of the PMU vs the CPU we're > running on, and this is already visible to userspace. > > Can't we just leave this responsibility to userspace? Believe me, I'm always a fan of offloading things to userspace :) If the VMM is privy to the details of the system it is on then the differing PMUs can be passed through to the guest w/ pinned vCPU threads. I just worry about the case of a naive VMM that assumes a homogenous system. I don't think I could entirely blame the VMM in this case either as we've gone to lengths to sanitise the feature set exposed to userspace. What happens when a vCPU gets scheduled on a core where the vPMU doesn't match? Ignoring other incongruences, it is not possible to virtualize more counters than are supported by the vPMU of the core. Stopping short of any major hacks in the kernel to fudge around the problem, I believe we may need to provide better documentation of how heterogeneous CPUs are handled in KVM and what userspace can do about it. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 54B39C25B4E for ; Fri, 20 Jan 2023 18:06:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EJj75YUdL4pUsG09QYiTzZJ2MVOC0TF2AD2uyaH1VwQ=; b=Kl18VcSNRHg8Y8 7VobZ8+/7pRzY17D+f/RZrJyePmd6ypCSO7H13Cn7NyrZr1A6UvR3WB5fZcmo2WLJRJvjHjV0IzHH GYY+cm6KR8xiuBs5JizpvxwVjwS52mxVTp3B/z5OXa30DKs269Jij8GE2GkCvV4BN+yJ2B8Yx9/BS VHWpgqW9Kp3FGr0Fk0iILfuIfMMf8OxwRQbwWhVDHXpr03lSY+TaWBc/A000M2CXuqCdmv7hrWLsc s4c8KxklYhDuf5rOl4DzackT/g3K4U2N38SqwsNCpyphHU1EtpMrYKYz1c7oqifR5uIRnxHnb4hdU wRVgtgHhLJ83qGBWZcWQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pIvlF-00BneF-Hp; Fri, 20 Jan 2023 18:05:09 +0000 Received: from out0.migadu.com ([2001:41d0:2:267::]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pIvlB-00Bnau-CX for linux-arm-kernel@lists.infradead.org; Fri, 20 Jan 2023 18:05:07 +0000 Date: Fri, 20 Jan 2023 18:04:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1674237899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Z1CaMcH4x5rtypgK1Xxe771oK8a9hWqV3gyDwwh42eA=; b=s+eRmBypvtugBDgcUxeYCjsvJV1Ae4KYybdsroHaliFb55C2HFYchZaA8jsQLjwHC/lmtn kYYWKySUBrw07RcLprP4ZxCmfxU8Td+Cx8vDrKSZdQohAxRDZfWRZdgW+sPyJILUzSDNXb EnmsfZCqROpfaQcSS24cv2UsT7n+gCs= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Marc Zyngier Cc: Reiji Watanabe , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, James Morse , Alexandru Elisei , Zenghui Yu , Suzuki K Poulose , Paolo Bonzini , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata Subject: Re: [PATCH v2 3/8] KVM: arm64: PMU: Preserve vCPU's PMCR_EL0.N value on vCPU reset Message-ID: References: <20230117013542.371944-1-reijiw@google.com> <20230117013542.371944-4-reijiw@google.com> <86pmb9mmkv.wl-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <86pmb9mmkv.wl-maz@kernel.org> X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230120_100506_023799_E0648530 X-CRM114-Status: GOOD ( 24.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hey Marc, On Fri, Jan 20, 2023 at 12:12:32PM +0000, Marc Zyngier wrote: > On Fri, 20 Jan 2023 00:30:33 +0000, Oliver Upton wrote: > > I think we need to derive a sanitised value for PMCR_EL0.N, as I believe > > nothing in the architecture prevents implementers from gluing together > > cores with varying numbers of PMCs. We probably haven't noticed it yet > > since it would appear all Arm designs have had 6 PMCs. > > This brings back the question of late onlining. How do you cope with > with the onlining of such a CPU that has a smaller set of counters > than its online counterparts? This is at odds with the way the PMU > code works. You're absolutely right, any illusion we derived from the online set of CPUs could fall apart with a late onlining of a different core. > If you have a different set of counters, you are likely to have a > different PMU altogether: > > [ 1.192606] hw perfevents: enabled with armv8_cortex_a57 PMU driver, 7 counters available > [ 1.201254] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available > > This isn't a broken system, but it has two set of cores which are > massively different, and two PMUs. > > This really should tie back to the PMU type we're counting on, and to > the set of CPUs that implements it. We already have some > infrastructure to check for the affinity of the PMU vs the CPU we're > running on, and this is already visible to userspace. > > Can't we just leave this responsibility to userspace? Believe me, I'm always a fan of offloading things to userspace :) If the VMM is privy to the details of the system it is on then the differing PMUs can be passed through to the guest w/ pinned vCPU threads. I just worry about the case of a naive VMM that assumes a homogenous system. I don't think I could entirely blame the VMM in this case either as we've gone to lengths to sanitise the feature set exposed to userspace. What happens when a vCPU gets scheduled on a core where the vPMU doesn't match? Ignoring other incongruences, it is not possible to virtualize more counters than are supported by the vPMU of the core. Stopping short of any major hacks in the kernel to fudge around the problem, I believe we may need to provide better documentation of how heterogeneous CPUs are handled in KVM and what userspace can do about it. -- Thanks, Oliver _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel