From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7CFC20EF for ; Fri, 20 Jan 2023 12:12:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7BC5CC433D2; Fri, 20 Jan 2023 12:12:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1674216755; bh=0iRYbwf/9VfNZDvMm3r5D5V9SjGnLfb8zDeKKD726pQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=dYnc6eLdwwBG1cYTK3l0gV197Jwvcv3DfcQMD7C3ZDWPUVN1nEQJ7kdFEdGf7ARlo H9oQJSIJ8mDcFW58izgVRn4Ex5lRA5IHko2t3aOhY3pNhfvvw9DyZIS9aVjEDcLiju rxUqJF4Ts+JJZqaGxNqN/sF5zObR3gUkYMGgLAGlfRFQvlaDLrpehQ0xeyCybAefIJ lzHLWa6rOCSPYU6ACvOfapi5fFmwJspzCfJOc7cUXoIzi7aa69dQCZCXcQxw8bzytO HRQpLRONAz3jYQfLGBIj/qKXAVvKfVtePF/i/jdpEHrq/Pu/AjtvY7gbtdtk2cPCZr 1zok2UNyqvTvw== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pIqG1-003P46-99; Fri, 20 Jan 2023 12:12:33 +0000 Date: Fri, 20 Jan 2023 12:12:32 +0000 Message-ID: <86pmb9mmkv.wl-maz@kernel.org> From: Marc Zyngier To: Oliver Upton Cc: Reiji Watanabe , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, James Morse , Alexandru Elisei , Zenghui Yu , Suzuki K Poulose , Paolo Bonzini , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata Subject: Re: [PATCH v2 3/8] KVM: arm64: PMU: Preserve vCPU's PMCR_EL0.N value on vCPU reset In-Reply-To: References: <20230117013542.371944-1-reijiw@google.com> <20230117013542.371944-4-reijiw@google.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: oliver.upton@linux.dev, reijiw@google.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, alexandru.elisei@arm.com, yuzenghui@huawei.com, suzuki.poulose@arm.com, pbonzini@redhat.com, ricarkol@google.com, jingzhangos@google.com, rananta@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Fri, 20 Jan 2023 00:30:33 +0000, Oliver Upton wrote: > > On Mon, Jan 16, 2023 at 05:35:37PM -0800, Reiji Watanabe wrote: > > The number of PMU event counters is indicated in PMCR_EL0.N. > > For a vCPU with PMUv3 configured, its value will be the same as > > the host value by default. Userspace can set PMCR_EL0.N for the > > vCPU to a lower value than the host value using KVM_SET_ONE_REG. > > However, it is practically unsupported, as reset_pmcr() resets > > PMCR_EL0.N to the host value on vCPU reset. > > > > Change reset_pmcr() to preserve the vCPU's PMCR_EL0.N value on > > vCPU reset so that userspace can limit the number of the PMU > > event counter on the vCPU. > > > > Signed-off-by: Reiji Watanabe > > --- > > arch/arm64/kvm/pmu-emul.c | 6 ++++++ > > arch/arm64/kvm/sys_regs.c | 4 +++- > > 2 files changed, 9 insertions(+), 1 deletion(-) > > > > diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c > > index 24908400e190..937a272b00a5 100644 > > --- a/arch/arm64/kvm/pmu-emul.c > > +++ b/arch/arm64/kvm/pmu-emul.c > > @@ -213,6 +213,12 @@ void kvm_pmu_vcpu_init(struct kvm_vcpu *vcpu) > > > > for (i = 0; i < ARMV8_PMU_MAX_COUNTERS; i++) > > pmu->pmc[i].idx = i; > > + > > + /* > > + * Initialize PMCR_EL0 for the vCPU with the host value so that > > + * the value is available at the very first vCPU reset. > > + */ > > + __vcpu_sys_reg(vcpu, PMCR_EL0) = read_sysreg(pmcr_el0); > > I think we need to derive a sanitised value for PMCR_EL0.N, as I believe > nothing in the architecture prevents implementers from gluing together > cores with varying numbers of PMCs. We probably haven't noticed it yet > since it would appear all Arm designs have had 6 PMCs. This brings back the question of late onlining. How do you cope with with the onlining of such a CPU that has a smaller set of counters than its online counterparts? This is at odds with the way the PMU code works. If you have a different set of counters, you are likely to have a different PMU altogether: [ 1.192606] hw perfevents: enabled with armv8_cortex_a57 PMU driver, 7 counters available [ 1.201254] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available This isn't a broken system, but it has two set of cores which are massively different, and two PMUs. This really should tie back to the PMU type we're counting on, and to the set of CPUs that implements it. We already have some infrastructure to check for the affinity of the PMU vs the CPU we're running on, and this is already visible to userspace. Can't we just leave this responsibility to userspace? Thanks, M. -- Without deviation from the norm, progress is not possible. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E1AACC25B50 for ; Fri, 20 Jan 2023 12:13:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=15iQA5ZMKOYxFS8WounF/xNL/nCf/0FTJaE8VZNHdmg=; b=4iINELcYJinQNU EgF04BEIkX/kLY+SR3NlqYs8IChWOT/hygzEqVcG20EsBZ7QXsVSKjWRMAfl1qJncOpAwqtMbITDE Hzr+ui85e8svQ0b18cKJ3JWQ1Po0lRttr4vAwsGCID4STL9STHGY1hCPMjbzI5ks6qsYpKTiij0nu 2wlksWNLjs+mdDMebAvxHa9Rz6iecOBi5/GDoK0gIosTsFkEGHjozbW/Xu70MvTGIZvH6RbDdmQLU tcqcNYM+ha5pExglb/dauiY7otXL0qZtmCDQynavCo0pNwwDd+W9Pfj6WLbPBrGwiPLYZp7bUD80W r8y2ZfBjTDiMEKPyG7Pw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pIqG7-00ACPo-7V; Fri, 20 Jan 2023 12:12:39 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pIqG4-00ACOh-ID for linux-arm-kernel@lists.infradead.org; Fri, 20 Jan 2023 12:12:38 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 19D8461F46; Fri, 20 Jan 2023 12:12:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7BC5CC433D2; Fri, 20 Jan 2023 12:12:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1674216755; bh=0iRYbwf/9VfNZDvMm3r5D5V9SjGnLfb8zDeKKD726pQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=dYnc6eLdwwBG1cYTK3l0gV197Jwvcv3DfcQMD7C3ZDWPUVN1nEQJ7kdFEdGf7ARlo H9oQJSIJ8mDcFW58izgVRn4Ex5lRA5IHko2t3aOhY3pNhfvvw9DyZIS9aVjEDcLiju rxUqJF4Ts+JJZqaGxNqN/sF5zObR3gUkYMGgLAGlfRFQvlaDLrpehQ0xeyCybAefIJ lzHLWa6rOCSPYU6ACvOfapi5fFmwJspzCfJOc7cUXoIzi7aa69dQCZCXcQxw8bzytO HRQpLRONAz3jYQfLGBIj/qKXAVvKfVtePF/i/jdpEHrq/Pu/AjtvY7gbtdtk2cPCZr 1zok2UNyqvTvw== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pIqG1-003P46-99; Fri, 20 Jan 2023 12:12:33 +0000 Date: Fri, 20 Jan 2023 12:12:32 +0000 Message-ID: <86pmb9mmkv.wl-maz@kernel.org> From: Marc Zyngier To: Oliver Upton Cc: Reiji Watanabe , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, James Morse , Alexandru Elisei , Zenghui Yu , Suzuki K Poulose , Paolo Bonzini , Ricardo Koller , Jing Zhang , Raghavendra Rao Anata Subject: Re: [PATCH v2 3/8] KVM: arm64: PMU: Preserve vCPU's PMCR_EL0.N value on vCPU reset In-Reply-To: References: <20230117013542.371944-1-reijiw@google.com> <20230117013542.371944-4-reijiw@google.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: oliver.upton@linux.dev, reijiw@google.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, alexandru.elisei@arm.com, yuzenghui@huawei.com, suzuki.poulose@arm.com, pbonzini@redhat.com, ricarkol@google.com, jingzhangos@google.com, rananta@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230120_041236_715120_717EC685 X-CRM114-Status: GOOD ( 35.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, 20 Jan 2023 00:30:33 +0000, Oliver Upton wrote: > > On Mon, Jan 16, 2023 at 05:35:37PM -0800, Reiji Watanabe wrote: > > The number of PMU event counters is indicated in PMCR_EL0.N. > > For a vCPU with PMUv3 configured, its value will be the same as > > the host value by default. Userspace can set PMCR_EL0.N for the > > vCPU to a lower value than the host value using KVM_SET_ONE_REG. > > However, it is practically unsupported, as reset_pmcr() resets > > PMCR_EL0.N to the host value on vCPU reset. > > > > Change reset_pmcr() to preserve the vCPU's PMCR_EL0.N value on > > vCPU reset so that userspace can limit the number of the PMU > > event counter on the vCPU. > > > > Signed-off-by: Reiji Watanabe > > --- > > arch/arm64/kvm/pmu-emul.c | 6 ++++++ > > arch/arm64/kvm/sys_regs.c | 4 +++- > > 2 files changed, 9 insertions(+), 1 deletion(-) > > > > diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c > > index 24908400e190..937a272b00a5 100644 > > --- a/arch/arm64/kvm/pmu-emul.c > > +++ b/arch/arm64/kvm/pmu-emul.c > > @@ -213,6 +213,12 @@ void kvm_pmu_vcpu_init(struct kvm_vcpu *vcpu) > > > > for (i = 0; i < ARMV8_PMU_MAX_COUNTERS; i++) > > pmu->pmc[i].idx = i; > > + > > + /* > > + * Initialize PMCR_EL0 for the vCPU with the host value so that > > + * the value is available at the very first vCPU reset. > > + */ > > + __vcpu_sys_reg(vcpu, PMCR_EL0) = read_sysreg(pmcr_el0); > > I think we need to derive a sanitised value for PMCR_EL0.N, as I believe > nothing in the architecture prevents implementers from gluing together > cores with varying numbers of PMCs. We probably haven't noticed it yet > since it would appear all Arm designs have had 6 PMCs. This brings back the question of late onlining. How do you cope with with the onlining of such a CPU that has a smaller set of counters than its online counterparts? This is at odds with the way the PMU code works. If you have a different set of counters, you are likely to have a different PMU altogether: [ 1.192606] hw perfevents: enabled with armv8_cortex_a57 PMU driver, 7 counters available [ 1.201254] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available This isn't a broken system, but it has two set of cores which are massively different, and two PMUs. This really should tie back to the PMU type we're counting on, and to the set of CPUs that implements it. We already have some infrastructure to check for the affinity of the PMU vs the CPU we're running on, and this is already visible to userspace. Can't we just leave this responsibility to userspace? Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel