From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFCC7182AA; Fri, 24 Nov 2023 12:51:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j3m3bFuo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A9F0C433C7; Fri, 24 Nov 2023 12:51:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700830300; bh=n0Ppb7Aek1/yhyaQZkBlGoo+37tHLU7EMg+F1uxGOjA=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=j3m3bFuo4rUyQrN1owEAUZA5fnueyTULAdpuBoF/36vKr1cmh3bCsWWkf89MUwG3K A5qE5xcG/1xB5REVGYMpC50mi4n8DV0H2caY4zYS/ENTtzGG1/TRo4Ake9f2Ywu6pf 3jJN3XhVIAYtSHlNHsS6w8MfE9Z4aznofzaueNUcgK35VveWzCYofmN8P1kJFuq/wy vT+PtM7c8X1vBBBYjLlMYaXuaa1HqEALbxI8xyTmRtrJ7JhtMMsXznJ/81mWwgz3Mr DkJa9fyR3r6kKT9YsJ1UdW297cknouSdrYl2LcmIAf2zNaO9dv/LaPxsujk8J0m6MU 5dxJussLgKadw== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1r6Vej-00G68i-V8; Fri, 24 Nov 2023 12:51:38 +0000 Date: Fri, 24 Nov 2023 12:51:36 +0000 Message-ID: <86o7fjco13.wl-maz@kernel.org> From: Marc Zyngier To: Ganapatrao Kulkarni Cc: Miguel Luis , "kvmarm@lists.linux.dev" , "kvm@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Darren Hart , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: Re: [PATCH v11 00/43] KVM: arm64: Nested Virtualization support (FEAT_NV2 only) In-Reply-To: <65dc2a93-0a17-4433-b3a5-430bf516ffe9@os.amperecomputing.com> References: <20231120131027.854038-1-maz@kernel.org> <86msv7ylnu.wl-maz@kernel.org> <05733774-4210-4097-9912-fb3aa8542fdd@oracle.com> <86a5r4zafh.wl-maz@kernel.org> <134912e4-beed-4ab6-8ce1-33e69ec382b3@os.amperecomputing.com> <868r6nzc5y.wl-maz@kernel.org> <65dc2a93-0a17-4433-b3a5-430bf516ffe9@os.amperecomputing.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gankulkarni@os.amperecomputing.com, miguel.luis@oracle.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, darren@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Fri, 24 Nov 2023 12:34:41 +0000, Ganapatrao Kulkarni wrote: >=20 >=20 >=20 > On 24-11-2023 03:49 pm, Marc Zyngier wrote: > > On Fri, 24 Nov 2023 09:50:33 +0000, > > Ganapatrao Kulkarni wrote: > >>=20 > >>=20 > >>=20 > >> On 23-11-2023 10:14 pm, Marc Zyngier wrote: > >>> On Thu, 23 Nov 2023 16:21:48 +0000, > >>> Miguel Luis wrote: > >>>>=20 > >>>> Hi Marc, > >>>>=20 > >>>> On 21/11/2023 18:02, Marc Zyngier wrote: > >>>>> On Tue, 21 Nov 2023 16:49:52 +0000, > >>>>> Miguel Luis wrote: > >>>>>> Hi Marc, > >>>>>>=20 > >>>>>>> On 20 Nov 2023, at 12:09, Marc Zyngier wrote: > >>>>>>>=20 > >>>>>>> This is the 5th drop of NV support on arm64 for this year, and mo= st > >>>>>>> probably the last one for this side of Christmas. > >>>>>>>=20 > >>>>>>> For the previous episodes, see [1]. > >>>>>>>=20 > >>>>>>> What's changed: > >>>>>>>=20 > >>>>>>> - Drop support for the original FEAT_NV. No existing hardware sup= ports > >>>>>>> it without FEAT_NV2, and the architecture is deprecating the f= ormer > >>>>>>> entirely. This results in fewer patches, and a slightly simpler > >>>>>>> model overall. > >>>>>>>=20 > >>>>>>> - Reorganise the series to make it a bit more logical now that FE= AT_NV > >>>>>>> is gone. > >>>>>>>=20 > >>>>>>> - Apply the NV idreg restrictions on VM first run rather than on = each > >>>>>>> access. > >>>>>>>=20 > >>>>>>> - Make the nested vgic shadow CPU interface a per-CPU structure r= ather > >>>>>>> than per-vcpu. > >>>>>>>=20 > >>>>>>> - Fix the EL0 timer fastpath > >>>>>>>=20 > >>>>>>> - Work around the architecture deficiencies when trapping WFI fro= m a > >>>>>>> L2 guest. > >>>>>>>=20 > >>>>>>> - Fix sampling of nested vgic state (MISR, ELRSR, EISR) > >>>>>>>=20 > >>>>>>> - Drop the patches that have already been merged (NV trap forward= ing, > >>>>>>> per-MMU VTCR) > >>>>>>>=20 > >>>>>>> - Rebased on top of 6.7-rc2 + the FEAT_E2H0 support [2]. > >>>>>>>=20 > >>>>>>> The branch containing these patches (and more) is at [3]. As for = the > >>>>>>> previous rounds, my intention is to take a prefix of this series = into > >>>>>>> 6.8, provided that it gets enough reviewing. > >>>>>>>=20 > >>>>>>> [1] https://lore.kernel.org/r/20230515173103.1017669-1-maz@kernel= .org > >>>>>>> [2] https://lore.kernel.org/r/20231120123721.851738-1-maz@kernel.= org > >>>>>>> [3] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platf= orms.git/log/?h=3Dkvm-arm64/nv-6.8-nv2-only > >>>>>>>=20 > >>>>>> While I was testing this with kvmtool for 5.16 I noted the followi= ng on dmesg: > >>>>>>=20 > >>>>>> [ 803.014258] kvm [19040]: Unsupported guest sys_reg access at: 8= 129fa50 [600003c9] > >>>>>> { Op0( 3), Op1( 5), CRn( 1), CRm( 0), Op2( 2), f= unc_read }, > >>>>>>=20 > >>>>>> This is CPACR_EL12. > >>>>> CPACR_EL12 is redirected to VNCR[0x100]. It really shouldn't trap... > >>>>>=20 > >>>>>> Still need yet to debug. > >>>>> Can you disassemble the guest around the offending PC? > >>>>=20 > >>>> [ 1248.686350] kvm [7013]: Unsupported guest sys_reg access at: 812b= aa50 [600003c9] > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 { Op0( 3), Op1( 5), CRn( 1), CRm( 0), Op2( 2), fun= c_read }, > >>>>=20 > >>>> =C2=A012baa00:=C2=A0=C2=A0=C2=A0 14000008 =C2=A0=C2=A0=C2=A0 b=C2= =A0=C2=A0=C2=A0 0x12baa20 > >>>> =C2=A012baa04:=C2=A0=C2=A0=C2=A0 d000d501 =C2=A0=C2=A0=C2=A0 adrp= =C2=A0=C2=A0=C2=A0 x1, 0x2d5c000 > >>>> =C2=A012baa08:=C2=A0=C2=A0=C2=A0 91154021 =C2=A0=C2=A0=C2=A0 add= =C2=A0=C2=A0=C2=A0 x1, x1, #0x550 > >>>> =C2=A012baa0c:=C2=A0=C2=A0=C2=A0 f9400022 =C2=A0=C2=A0=C2=A0 ldr= =C2=A0=C2=A0=C2=A0 x2, [x1] > >>>> =C2=A012baa10:=C2=A0=C2=A0=C2=A0 f9400421 =C2=A0=C2=A0=C2=A0 ldr= =C2=A0=C2=A0=C2=A0 x1, [x1, #8] > >>>> =C2=A012baa14:=C2=A0=C2=A0=C2=A0 8a010042 =C2=A0=C2=A0=C2=A0 and= =C2=A0=C2=A0=C2=A0 x2, x2, x1 > >>>> =C2=A012baa18:=C2=A0=C2=A0=C2=A0 d3441c42 =C2=A0=C2=A0=C2=A0 ubfx= =C2=A0=C2=A0=C2=A0 x2, x2, #4, #4 > >>>> =C2=A012baa1c:=C2=A0=C2=A0=C2=A0 b4000082 =C2=A0=C2=A0=C2=A0 cbz= =C2=A0=C2=A0=C2=A0 x2, 0x12baa2c > >>>> =C2=A012baa20:=C2=A0=C2=A0=C2=A0 d2a175a0 =C2=A0=C2=A0=C2=A0 mov= =C2=A0=C2=A0=C2=A0 x0, #0xbad0000=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 // #195887104 > >>>> =C2=A012baa24:=C2=A0=C2=A0=C2=A0 f2994220 =C2=A0=C2=A0=C2=A0 movk= =C2=A0=C2=A0=C2=A0 x0, #0xca11 > >>>> =C2=A012baa28:=C2=A0=C2=A0=C2=A0 d69f03e0 =C2=A0=C2=A0=C2=A0 eret > >>>> =C2=A012baa2c:=C2=A0=C2=A0=C2=A0 d2c00080 =C2=A0=C2=A0=C2=A0 mov= =C2=A0=C2=A0=C2=A0 x0, #0x400000000=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 // #17179869184 > >>>> =C2=A012baa30:=C2=A0=C2=A0=C2=A0 f2b10000 =C2=A0=C2=A0=C2=A0 movk= =C2=A0=C2=A0=C2=A0 x0, #0x8800, lsl #16 > >>>> =C2=A012baa34:=C2=A0=C2=A0=C2=A0 f2800000 =C2=A0=C2=A0=C2=A0 movk= =C2=A0=C2=A0=C2=A0 x0, #0x0 > >>>> =C2=A012baa38:=C2=A0=C2=A0=C2=A0 d51c1100 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 hcr_el2, x0 > >>>> =C2=A012baa3c:=C2=A0=C2=A0=C2=A0 d5033fdf =C2=A0=C2=A0=C2=A0 isb ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This. > >>>> =C2=A012baa40:=C2=A0=C2=A0=C2=A0 d53c4100 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, sp_el1 > >>>> =C2=A012baa44:=C2=A0=C2=A0=C2=A0 9100001f =C2=A0=C2=A0=C2=A0 mov= =C2=A0=C2=A0=C2=A0 sp, x0 > >>>> =C2=A012baa48:=C2=A0=C2=A0=C2=A0 d538d080 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, tpidr_el1 > >>>> =C2=A012baa4c:=C2=A0=C2=A0=C2=A0 d51cd040 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 tpidr_el2, x0 > >>>> =C2=A012baa50:=C2=A0=C2=A0=C2=A0 d53d1040 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, cpacr_el12 > >>>> =C2=A012baa54:=C2=A0=C2=A0=C2=A0 d5181040 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 cpacr_el1, x0 > >>>> =C2=A012baa58:=C2=A0=C2=A0=C2=A0 d53dc000 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, vbar_el12 > >>>> =C2=A012baa5c:=C2=A0=C2=A0=C2=A0 d518c000 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 vbar_el1, x0 > >>>> =C2=A012baa60:=C2=A0=C2=A0=C2=A0 d53c1120 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, mdcr_el2 > >>>> =C2=A012baa64:=C2=A0=C2=A0=C2=A0 9272f400 =C2=A0=C2=A0=C2=A0 and= =C2=A0=C2=A0=C2=A0 x0, x0, #0xffffffffffffcfff > >>>> =C2=A012baa68:=C2=A0=C2=A0=C2=A0 9266f400 =C2=A0=C2=A0=C2=A0 and= =C2=A0=C2=A0=C2=A0 x0, x0, #0xfffffffffcffffff > >>>> =C2=A012baa6c:=C2=A0=C2=A0=C2=A0 d51c1120 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 mdcr_el2, x0 > >>>> =C2=A012baa70:=C2=A0=C2=A0=C2=A0 d53d2040 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, tcr_el12 > >>>> =C2=A012baa74:=C2=A0=C2=A0=C2=A0 d5182040 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 tcr_el1, x0 > >>>> =C2=A012baa78:=C2=A0=C2=A0=C2=A0 d53d2000 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, ttbr0_el12 > >>>> =C2=A012baa7c:=C2=A0=C2=A0=C2=A0 d5182000 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 ttbr0_el1, x0 > >>>> =C2=A012baa80:=C2=A0=C2=A0=C2=A0 d53d2020 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, ttbr1_el12 > >>>> =C2=A012baa84:=C2=A0=C2=A0=C2=A0 d5182020 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 ttbr1_el1, x0 > >>>> =C2=A012baa88:=C2=A0=C2=A0=C2=A0 d53da200 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, mair_el12 > >>>> =C2=A012baa8c:=C2=A0=C2=A0=C2=A0 d518a200 =C2=A0=C2=A0=C2=A0 msr= =C2=A0=C2=A0=C2=A0 mair_el1, x0 > >>>> =C2=A012baa90:=C2=A0=C2=A0=C2=A0 d5380761 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x1, s3_0_c0_c7_3 > >>>> =C2=A012baa94:=C2=A0=C2=A0=C2=A0 d3400c21 =C2=A0=C2=A0=C2=A0 ubfx= =C2=A0=C2=A0=C2=A0 x1, x1, #0, #4 > >>>> =C2=A012baa98:=C2=A0=C2=A0=C2=A0 b4000141 =C2=A0=C2=A0=C2=A0 cbz= =C2=A0=C2=A0=C2=A0 x1, 0x12baac0 > >>>> =C2=A012baa9c:=C2=A0=C2=A0=C2=A0 d53d2060 =C2=A0=C2=A0=C2=A0 mrs= =C2=A0=C2=A0=C2=A0 x0, s3_5_c2_c0_3 > >>>=20 > >>> OK, this is suspiciously close to the location Ganapatrao was having > >>> issues with. Are you running on the same hardware? > >>>=20 > >>> In any case, we should never take a trap for this access. Can you dump > >>> HCR_EL2 at the point where the guest traps (in switch.c)? > >>>=20 > >>=20 > >> I have dumped HCR_EL2 before entry to L1 in both V11 and V10. > >> on V10 HCR_EL2=3D0x2743c827c263f > >> on V11 HCR_EL2=3D0x27c3c827c263f > >>=20 > >> on V11 the function vcpu_el2_e2h_is_set(vcpu) is returning false > >> resulting in NV1 bit set along with NV and NV2. > >> AFAIK, For L1 to be in VHE, NV1 bit should be zero and NV=3DNV2=3D1. > >>=20 > >> I could boot L1 then L2, if I hack vcpu_el2_e2h_is_set to return true. > >> There could be a bug in V11 or E2H0 patchset resulting in > >> vcpu_el2_e2h_is_set() returning false? > >=20 > > The E2H0 series should only force vcpu_el2_e2h_is_set() to return > > true, but not set it to false. Can you dump the *guest's* version of > > HCR_EL2 at this point? > >=20 >=20 > with V11: vhcr_el2=3D0x100030080000000 mask=3D0x100af00ffffffff How is this value possible if the write to HCR_EL2 has taken place? When do you sample this? > with V10: vhcr_el2=3D0x488000000 > with hack+V11: vhcr_el2=3D0x488000000 mask=3D0x100af00ffffffff Well, of course, if you constrain the value of HCR_EL2... M. --=20 Without deviation from the norm, progress is not possible.