From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 102BEDDA6; Mon, 27 Nov 2023 09:22:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OkT50YF3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8843AC433C8; Mon, 27 Nov 2023 09:22:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1701076923; bh=BOixlSNocNhyGWuF5ccWPK61QKCpJuHHl7gm2DfKVQ0=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=OkT50YF326qn4QfUxo1EAy8lrIdwkp6UZ4JN85QzYZoXpPuivM1BAyRzZXeNHT3Ds KTh17jZ8DhEIoS7Ee6rgpT+GKrxjbRs6Jc2YbHWEmE4VWNwlahpIveDDR8oDhO0J2E 4DdT4lkpa+LAqurxKMJNwXK4dV8PERZZOHyFj96i9KOfKKpfdndRtQTEyOqbFLE50l sJSE9KtfeQYWktD+vT41bMa/FkdzbHwHpz7g9VHvjeJQTW9gEc7HSeaeEDUKK+TBDd oIsPYJgJHHiQzE0C/3HxojJLGELew8sTjhZDFVqIc46r1En0tpi6oVZZ1XIqpV9/Z4 09H26visiFYoQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1r7XoW-00Gj5n-TY; Mon, 27 Nov 2023 09:22:01 +0000 Date: Mon, 27 Nov 2023 09:22:00 +0000 Message-ID: <86jzq3d007.wl-maz@kernel.org> From: Marc Zyngier To: Ganapatrao Kulkarni Cc: Miguel Luis , "kvmarm@lists.linux.dev" , "kvm@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Darren Hart , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: Re: [PATCH v11 00/43] KVM: arm64: Nested Virtualization support (FEAT_NV2 only) In-Reply-To: <9f8656c7-8036-4bd6-a0f5-4fa531f95b2f@os.amperecomputing.com> References: <20231120131027.854038-1-maz@kernel.org> <86msv7ylnu.wl-maz@kernel.org> <05733774-4210-4097-9912-fb3aa8542fdd@oracle.com> <86a5r4zafh.wl-maz@kernel.org> <134912e4-beed-4ab6-8ce1-33e69ec382b3@os.amperecomputing.com> <868r6nzc5y.wl-maz@kernel.org> <65dc2a93-0a17-4433-b3a5-430bf516ffe9@os.amperecomputing.com> <86o7fjco13.wl-maz@kernel.org> <86leancjcr.wl-maz@kernel.org> <9f8656c7-8036-4bd6-a0f5-4fa531f95b2f@os.amperecomputing.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gankulkarni@os.amperecomputing.com, miguel.luis@oracle.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, darren@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Mon, 27 Nov 2023 07:26:58 +0000, Ganapatrao Kulkarni wrote: > > > > On 24-11-2023 08:02 pm, Marc Zyngier wrote: > > On Fri, 24 Nov 2023 13:22:22 +0000, > > Ganapatrao Kulkarni wrote: > >> > >>> How is this value possible if the write to HCR_EL2 has taken place? > >>> When do you sample this? > >> > >> I am not sure how and where it got set. I think, whatever it is set, > >> it is due to false return of vcpu_el2_e2h_is_set(). Need to > >> understand/debug. > >> The vhcr_el2 value I have shared is traced along with hcr in function > >> __activate_traps/__compute_hcr. > > > > Here's my hunch: > > > > The guest boots with E2H=0, because we don't advertise anything else > > on your HW. So we run with NV1=1 until we try to *upgrade* to VHE. NV2 > > means that HCR_EL2 is writable (to memory) without a trap. But we're > > still running with NV1=1. > > > > Subsequently, we access a sysreg that should never trap for a VHE > > guest, but we're with the wrong config. Bad things happen. > > > > Unfortunately, NV2 is pretty much incompatible with E2H being updated, > > because it cannot perform the changes that this would result into at > > the point where they should happen. We can try and do a best effort > > handling, but you can always trick it. > > > > Anyway, can you see if the hack below helps? I'm not keen on it at > > all, but this would be a good data point. > > Thanks Marc, this diff fixes the issue. > Just wondering what is changed w.r.t to L1 handling from V10 to V11 > that it requires this trick? Not completely sure. Before v11, anything that would trap would be silently handled by the FEAT_NV code. Now, a trap for something that is supposed to be redirected to VNCR results in an UNDEF exception. I suspect that the exception is handled again as a call to __finalise_el2(), probably because the write to VBAR_EL1 didn't do what it was supposed to do. > Also why this was not seen on your platform, is it E2H0 enabled? It doesn't have FEAT_E2H0, and that's the whole point. No E2H0, no problems, as the guest cannot trick the host into losing track of the state (which I'm pretty sure can happen even with this ugly hack). I will probably completely disable NV1 support in the next drop, and make NV support only VHE guests. Which is the only mode that makes any sense anyway. Thanks, M. -- Without deviation from the norm, progress is not possible.