From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 841BBC433C1 for ; Tue, 30 Mar 2021 09:57:37 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D059D61999 for ; Tue, 30 Mar 2021 09:57:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D059D61999 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Subject:Cc:To: From:Message-ID:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=j/7E2kfCo8PVni/e3O/Qsdpa1nvfNiQig2o2GEQvomY=; b=LLO9UZpDNJyl9T/9jPs3d/kSz LXFBeygdDb+UfTJirHfsRk8jJdkN/tNiinhreQnIoncSNaGvogsuaV6HHSf+ywI16IjZAzxkPWCz3 /YBiz2/12MqWQwVUU5l4EDAbevwGWjT9dCfMmxqH4uitR5qovqJEvr8cBO5N8DoBnqG84aQKnakcV VAUdY8oE/qwTrY8fO67UZaBejZH2Wby66SDNOVaD7lkurx/bet420z9a3+Hi2ZleJQGFFeFOkXfLt fWd9olpw6sNxNbrQSJda9vpOxLpRHUmDIxvb76XP7kMmgMP4v9t/bOltcuWstnvx3fdXKT87nUwGz oWsjOkyrg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lRB65-003JYQ-6e; Tue, 30 Mar 2021 09:55:41 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lRB5Z-003JTl-W8 for linux-arm-kernel@lists.infradead.org; Tue, 30 Mar 2021 09:55:12 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 35E2B6157F; Tue, 30 Mar 2021 09:55:08 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1lRB5W-004e60-0a; Tue, 30 Mar 2021 10:55:06 +0100 Date: Tue, 30 Mar 2021 10:55:04 +0100 Message-ID: <87sg4dq83r.wl-maz@kernel.org> From: Marc Zyngier To: Alexandru Elisei Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Subject: Re: [PATCH v2] KVM: arm64: Initialize VCPU mdcr_el2 before loading it In-Reply-To: <20210323180057.263356-1-alexandru.elisei@arm.com> References: <20210323180057.263356-1-alexandru.elisei@arm.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: alexandru.elisei@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210330_105510_627928_20E8DE79 X-CRM114-Status: GOOD ( 51.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Alex, On Tue, 23 Mar 2021 18:00:57 +0000, Alexandru Elisei wrote: > > When a VCPU is created, the kvm_vcpu struct is initialized to zero in > kvm_vm_ioctl_create_vcpu(). On VHE systems, the first time > vcpu.arch.mdcr_el2 is loaded on hardware is in vcpu_load(), before it is > set to a sensible value in kvm_arm_setup_debug() later in the run loop. The > result is that KVM executes for a short time with MDCR_EL2 set to zero. > > This has several unintended consequences: > > * Setting MDCR_EL2.HPMN to 0 is constrained unpredictable according to ARM > DDI 0487G.a, page D13-3820. The behavior specified by the architecture > in this case is for the PE to behave as if MDCR_EL2.HPMN is set to a > value less than or equal to PMCR_EL0.N, which means that an unknown > number of counters are now disabled by MDCR_EL2.HPME, which is zero. > > * The host configuration for the other debug features controlled by > MDCR_EL2 is temporarily lost. This has been harmless so far, as Linux > doesn't use the other fields, but that might change in the future. > > Let's avoid both issues by initializing the VCPU's mdcr_el2 field in > kvm_vcpu_vcpu_first_run_init(), thus making sure that the MDCR_EL2 register > has a consistent value after each vcpu_load(). > > Signed-off-by: Alexandru Elisei This looks strangely similar to 4942dc6638b0 ("KVM: arm64: Write arch.mdcr_el2 changes since last vcpu_load on VHE"), just at a different point. Probably worth a Fixes tag. > --- > Found by code inspection. Based on v5.12-rc4. > > Tested on an odroid-c4 with VHE. vcpu->arch.mdcr_el2 is calculated to be > 0x4e66. Without this patch, reading MDCR_EL2 after the first vcpu_load() in > kvm_arch_vcpu_ioctl_run() returns 0; with this patch it returns the correct > value, 0xe66 (FEAT_SPE is not implemented by the PE). > > This patch was initially part of the KVM SPE series [1], but those patches > haven't seen much activity, so I thought it would be a good idea to send > this patch separately to draw more attention to it. > > Changes in v2: > * Moved kvm_arm_vcpu_init_debug() earlier in kvm_vcpu_first_run_init() so > vcpu->arch.mdcr_el2 is calculated even if kvm_vgic_map_resources() fails. > * Added comment to kvm_arm_setup_mdcr_el2 to explain what testing > vcpu->guest_debug means. > > [1] https://www.spinics.net/lists/kvm-arm/msg42959.html > > arch/arm64/include/asm/kvm_host.h | 1 + > arch/arm64/kvm/arm.c | 3 +- > arch/arm64/kvm/debug.c | 82 +++++++++++++++++++++---------- > 3 files changed, 59 insertions(+), 27 deletions(-) > > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h > index 3d10e6527f7d..858c2fcfc043 100644 > --- a/arch/arm64/include/asm/kvm_host.h > +++ b/arch/arm64/include/asm/kvm_host.h > @@ -713,6 +713,7 @@ static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {} > static inline void kvm_arch_vcpu_block_finish(struct kvm_vcpu *vcpu) {} > > void kvm_arm_init_debug(void); > +void kvm_arm_vcpu_init_debug(struct kvm_vcpu *vcpu); > void kvm_arm_setup_debug(struct kvm_vcpu *vcpu); > void kvm_arm_clear_debug(struct kvm_vcpu *vcpu); > void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu); > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c > index 7f06ba76698d..7088d8fe7186 100644 > --- a/arch/arm64/kvm/arm.c > +++ b/arch/arm64/kvm/arm.c > @@ -580,6 +580,8 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu) > > vcpu->arch.has_run_once = true; > > + kvm_arm_vcpu_init_debug(vcpu); > + > if (likely(irqchip_in_kernel(kvm))) { > /* > * Map the VGIC hardware resources before running a vcpu the > @@ -791,7 +793,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) > } > > kvm_arm_setup_debug(vcpu); > - Spurious change? > /************************************************************** > * Enter the guest > */ > diff --git a/arch/arm64/kvm/debug.c b/arch/arm64/kvm/debug.c > index 7a7e425616b5..3626d03354f6 100644 > --- a/arch/arm64/kvm/debug.c > +++ b/arch/arm64/kvm/debug.c > @@ -68,6 +68,60 @@ void kvm_arm_init_debug(void) > __this_cpu_write(mdcr_el2, kvm_call_hyp_ret(__kvm_get_mdcr_el2)); > } > > +/** > + * kvm_arm_setup_mdcr_el2 - configure vcpu mdcr_el2 value > + * > + * @vcpu: the vcpu pointer > + * @host_mdcr: host mdcr_el2 value > + * > + * This ensures we will trap access to: > + * - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) > + * - Debug ROM Address (MDCR_EL2_TDRA) > + * - OS related registers (MDCR_EL2_TDOSA) > + * - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) > + */ > +static void kvm_arm_setup_mdcr_el2(struct kvm_vcpu *vcpu, u32 host_mdcr) > +{ > + bool trap_debug = !(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY); > + > + /* > + * This also clears MDCR_EL2_E2PB_MASK to disable guest access > + * to the profiling buffer. > + */ > + vcpu->arch.mdcr_el2 = host_mdcr & MDCR_EL2_HPMN_MASK; > + vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | > + MDCR_EL2_TPMS | > + MDCR_EL2_TPMCR | > + MDCR_EL2_TDRA | > + MDCR_EL2_TDOSA); > + > + /* Is the VM being debugged by userspace? */ > + if (vcpu->guest_debug) { > + /* Route all software debug exceptions to EL2 */ > + vcpu->arch.mdcr_el2 |= MDCR_EL2_TDE; > + if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW) > + trap_debug = true; > + } > + > + /* Trap debug register access */ > + if (trap_debug) > + vcpu->arch.mdcr_el2 |= MDCR_EL2_TDA; > + > + trace_kvm_arm_set_dreg32("MDCR_EL2", vcpu->arch.mdcr_el2); > +} > + > +/** > + * kvm_arm_vcpu_init_debug - setup vcpu debug traps > + * > + * @vcpu: the vcpu pointer > + * > + * Set vcpu initial mdcr_el2 value. > + */ > +void kvm_arm_vcpu_init_debug(struct kvm_vcpu *vcpu) > +{ > + kvm_arm_setup_mdcr_el2(vcpu, this_cpu_read(mdcr_el2)); Given that kvm_arm_setup_mdcr_el2() always takes the current host value for mdcr_el2, why not moving the read into it and be done with it? Also, do we really need an extra wrapper? > +} > + > /** > * kvm_arm_reset_debug_ptr - reset the debug ptr to point to the vcpu state > */ > @@ -83,12 +137,7 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu) > * @vcpu: the vcpu pointer > * > * This is called before each entry into the hypervisor to setup any > - * debug related registers. Currently this just ensures we will trap > - * access to: > - * - Performance monitors (MDCR_EL2_TPM/MDCR_EL2_TPMCR) > - * - Debug ROM Address (MDCR_EL2_TDRA) > - * - OS related registers (MDCR_EL2_TDOSA) > - * - Statistical profiler (MDCR_EL2_TPMS/MDCR_EL2_E2PB) > + * debug related registers. > * > * Additionally, KVM only traps guest accesses to the debug registers if > * the guest is not actively using them (see the KVM_ARM64_DEBUG_DIRTY > @@ -100,27 +149,14 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu) > > void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) > { > - bool trap_debug = !(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY); > unsigned long mdscr, orig_mdcr_el2 = vcpu->arch.mdcr_el2; > > trace_kvm_arm_setup_debug(vcpu, vcpu->guest_debug); > > - /* > - * This also clears MDCR_EL2_E2PB_MASK to disable guest access > - * to the profiling buffer. > - */ > - vcpu->arch.mdcr_el2 = __this_cpu_read(mdcr_el2) & MDCR_EL2_HPMN_MASK; > - vcpu->arch.mdcr_el2 |= (MDCR_EL2_TPM | > - MDCR_EL2_TPMS | > - MDCR_EL2_TPMCR | > - MDCR_EL2_TDRA | > - MDCR_EL2_TDOSA); > + kvm_arm_setup_mdcr_el2(vcpu, __this_cpu_read(mdcr_el2)); > > /* Is Guest debugging in effect? */ > if (vcpu->guest_debug) { > - /* Route all software debug exceptions to EL2 */ > - vcpu->arch.mdcr_el2 |= MDCR_EL2_TDE; > - > /* Save guest debug state */ > save_guest_debug_regs(vcpu); > > @@ -174,7 +210,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) > > vcpu->arch.debug_ptr = &vcpu->arch.external_debug_state; > vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; > - trap_debug = true; There is something that slightly worries me here: there is now a disconnect between flagging debug as dirty and setting the trapping. And actually, you now check for KVM_ARM64_DEBUG_DIRTY and set the trap bits *before* setting the dirty bit itself. Here, I believe you end up with guest/host confusion of breakpoints, which isn't great. Or did I miss something? > > trace_kvm_arm_set_regset("BKPTS", get_num_brps(), > &vcpu->arch.debug_ptr->dbg_bcr[0], > @@ -189,10 +224,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) > BUG_ON(!vcpu->guest_debug && > vcpu->arch.debug_ptr != &vcpu->arch.vcpu_debug_state); > > - /* Trap debug register access */ > - if (trap_debug) > - vcpu->arch.mdcr_el2 |= MDCR_EL2_TDA; > - > /* If KDE or MDE are set, perform a full save/restore cycle. */ > if (vcpu_read_sys_reg(vcpu, MDSCR_EL1) & (DBG_MDSCR_KDE | DBG_MDSCR_MDE)) > vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; > @@ -201,7 +232,6 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) > if (has_vhe() && orig_mdcr_el2 != vcpu->arch.mdcr_el2) > write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2); > > - trace_kvm_arm_set_dreg32("MDCR_EL2", vcpu->arch.mdcr_el2); > trace_kvm_arm_set_dreg32("MDSCR_EL1", vcpu_read_sys_reg(vcpu, MDSCR_EL1)); > } Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel