From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A17CEE49A5 for ; Tue, 22 Aug 2023 06:49:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Do7W5cTQoUNDODyto2fdSnqU9+zC/hbd4te/wei9LXE=; b=GGnXpV5MuJHe9E 4G0E0TbAvy1lN319A1IvRZj8zrJkSPavOLtNy6cWHveGn5xSdU1sL0S9xwKFmGL3l7A4qu0TJT3cv JyI1HGr0UW2s/nK2cNOvTdwjppudWTzUqmmU31Bnz0q5KpWcFYjQ/7TPyC5o6d+hHDn8SrrpS47Uq MkaFzg5xLH/TNMGZwMibcZg04h5t2FCWvNuDz2EU5E+/dYvxzH5hrbTkzpoamw43zUbYlpt/ZrmBK Up9Wfm/sBeo6BM3L7pz+zUk0qIPF1+bIUFy37uj5w/KcRNruB/l1yrvsG5BSLKAdC4UQ0dN7UNnJr znpUN7dWLjoa/lWGRAmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qYLBx-00FGEw-0A; Tue, 22 Aug 2023 06:48:41 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qYLBt-00FGBH-23 for linux-arm-kernel@lists.infradead.org; Tue, 22 Aug 2023 06:48:39 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B64A32F4; Mon, 21 Aug 2023 23:49:13 -0700 (PDT) Received: from FVFF77S0Q05N (unknown [10.57.3.221]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 492343F64C; Mon, 21 Aug 2023 23:48:31 -0700 (PDT) Date: Tue, 22 Aug 2023 07:48:28 +0100 From: Mark Rutland To: Marc Zyngier , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Huang Shijie , Leo Yan Subject: Re: [PATCH v2] KVM: arm64: pmu: Resync EL0 state on counter rotation Message-ID: References: <20230820090108.177817-1-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20230820090108.177817-1-maz@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230821_234837_776236_E621AF69 X-CRM114-Status: GOOD ( 29.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Sun, Aug 20, 2023 at 10:01:08AM +0100, Marc Zyngier wrote: > Huang Shijie reports that, when profiling a guest from the host > with a number of events that exceeds the number of available > counters, the reported counts are wildly inaccurate. Without > the counter oversubscription, the reported counts are correct. > > Their investigation indicates that upon counter rotation (which > takes place on the back of a timer interrupt), we fail to > re-apply the guest EL0 enabling, leading to the counting of host > events instead of guest events. > > In order to solve this, add yet another hook between the host PMU > driver and KVM, re-applying the guest EL0 configuration if the > right conditions apply (the host is VHE, we are in interrupt > context, and we interrupted a running vcpu). This triggers a new > vcpu request which will apply the correct configuration on guest > reentry. > > With this, we have the correct counts, even when the counters are > oversubscribed. > > Reported-by: Huang Shijie > Suggested-by: Oliver Upton > Tested_by: Huang Shijie > Signed-off-by: Marc Zyngier > Cc: Leo Yan > Cc: Mark Rutland > Cc: Will Deacon > Link: https://lore.kernel.org/r/20230809013953.7692-1-shijie@os.amperecomputing.com This looks sane to me so: Acked-by: Mark Rutland It doesn't look like we have any PMU patches queued that would conflict, so if Will's happy I reckon this should go through the KVM/arm64 tree. Mark. > --- > > Notes: > V2: Fixed 32bit compilation > > arch/arm/include/asm/arm_pmuv3.h | 2 ++ > arch/arm64/include/asm/kvm_host.h | 1 + > arch/arm64/kvm/arm.c | 3 +++ > arch/arm64/kvm/pmu.c | 18 ++++++++++++++++++ > drivers/perf/arm_pmuv3.c | 2 ++ > include/kvm/arm_pmu.h | 2 ++ > 6 files changed, 28 insertions(+) > > diff --git a/arch/arm/include/asm/arm_pmuv3.h b/arch/arm/include/asm/arm_pmuv3.h > index f3cd04ff022d..72529f5e2bed 100644 > --- a/arch/arm/include/asm/arm_pmuv3.h > +++ b/arch/arm/include/asm/arm_pmuv3.h > @@ -227,6 +227,8 @@ static inline bool kvm_set_pmuserenr(u64 val) > return false; > } > > +static inline void kvm_vcpu_pmu_resync_el0(void) {} > + > /* PMU Version in DFR Register */ > #define ARMV8_PMU_DFR_VER_NI 0 > #define ARMV8_PMU_DFR_VER_V3P4 0x5 > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h > index d3dd05bbfe23..553040e0e375 100644 > --- a/arch/arm64/include/asm/kvm_host.h > +++ b/arch/arm64/include/asm/kvm_host.h > @@ -49,6 +49,7 @@ > #define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) > #define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5) > #define KVM_REQ_SUSPEND KVM_ARCH_REQ(6) > +#define KVM_REQ_RESYNC_PMU_EL0 KVM_ARCH_REQ(7) > > #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ > KVM_DIRTY_LOG_INITIALLY_SET) > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c > index 72dc53a75d1c..978b0411082f 100644 > --- a/arch/arm64/kvm/arm.c > +++ b/arch/arm64/kvm/arm.c > @@ -803,6 +803,9 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu) > kvm_pmu_handle_pmcr(vcpu, > __vcpu_sys_reg(vcpu, PMCR_EL0)); > > + if (kvm_check_request(KVM_REQ_RESYNC_PMU_EL0, vcpu)) > + kvm_vcpu_pmu_restore_guest(vcpu); > + > if (kvm_check_request(KVM_REQ_SUSPEND, vcpu)) > return kvm_vcpu_suspend(vcpu); > > diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c > index 121f1a14c829..0eea225fd09a 100644 > --- a/arch/arm64/kvm/pmu.c > +++ b/arch/arm64/kvm/pmu.c > @@ -236,3 +236,21 @@ bool kvm_set_pmuserenr(u64 val) > ctxt_sys_reg(hctxt, PMUSERENR_EL0) = val; > return true; > } > + > +/* > + * If we interrupted the guest to update the host PMU context, make > + * sure we re-apply the guest EL0 state. > + */ > +void kvm_vcpu_pmu_resync_el0(void) > +{ > + struct kvm_vcpu *vcpu; > + > + if (!has_vhe() || !in_interrupt()) > + return; > + > + vcpu = kvm_get_running_vcpu(); > + if (!vcpu) > + return; > + > + kvm_make_request(KVM_REQ_RESYNC_PMU_EL0, vcpu); > +} > diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c > index 08b3a1bf0ef6..6a3d8176f54a 100644 > --- a/drivers/perf/arm_pmuv3.c > +++ b/drivers/perf/arm_pmuv3.c > @@ -772,6 +772,8 @@ static void armv8pmu_start(struct arm_pmu *cpu_pmu) > > /* Enable all counters */ > armv8pmu_pmcr_write(armv8pmu_pmcr_read() | ARMV8_PMU_PMCR_E); > + > + kvm_vcpu_pmu_resync_el0(); > } > > static void armv8pmu_stop(struct arm_pmu *cpu_pmu) > diff --git a/include/kvm/arm_pmu.h b/include/kvm/arm_pmu.h > index 847da6fc2713..3a8a70a60794 100644 > --- a/include/kvm/arm_pmu.h > +++ b/include/kvm/arm_pmu.h > @@ -74,6 +74,7 @@ int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu); > struct kvm_pmu_events *kvm_get_pmu_events(void); > void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu); > void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu); > +void kvm_vcpu_pmu_resync_el0(void); > > #define kvm_vcpu_has_pmu(vcpu) \ > (test_bit(KVM_ARM_VCPU_PMU_V3, (vcpu)->arch.features)) > @@ -171,6 +172,7 @@ static inline u8 kvm_arm_pmu_get_pmuver_limit(void) > { > return 0; > } > +static inline void kvm_vcpu_pmu_resync_el0(void) {} > > #endif > > -- > 2.39.2 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel