From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6B62199B3; Fri, 24 Nov 2023 10:19:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="szTiw9F2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B394C433C8; Fri, 24 Nov 2023 10:19:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700821164; bh=ezJrSp23KXb/hO6ks68IIX+fbvqWMNg+9nVoAxoKW8Q=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=szTiw9F2cGc2s71ZDdtpxoFF0EjWH2OY7N5AfnoWEE8zubiqo3Zn+JxLg7za7mO0p t1HV088SZOdi7zVKtTSwy2B/WJtDOXUyHQ4gLpzWm+9gi5B9Ih1rTTy1D0SdPOpwtT y0vjOpAuo08SS4pEyN2RL8Neyoj53swr+5je3F6etgitdJmzihA6SJntl/04QpCTQk RlzJ5ewHqXd/cpcRESrgv8emSxVUSEG34zj96iIOgcjK0L9G/lA7pBibiobDA1uryw pHTGB3TDAkEbxK+ulSe/wZFM5BC9vOZtQfYrdZYegvXd3tjM2KlCxsbXtwtU899BXC BHcf7Bh+ddkwg== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1r6THN-00G3Ie-Jc; Fri, 24 Nov 2023 10:19:21 +0000 Date: Fri, 24 Nov 2023 10:19:21 +0000 Message-ID: <868r6nzc5y.wl-maz@kernel.org> From: Marc Zyngier To: Ganapatrao Kulkarni Cc: Miguel Luis , "kvmarm@lists.linux.dev" , "kvm@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Darren Hart , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: Re: [PATCH v11 00/43] KVM: arm64: Nested Virtualization support (FEAT_NV2 only) In-Reply-To: <134912e4-beed-4ab6-8ce1-33e69ec382b3@os.amperecomputing.com> References: <20231120131027.854038-1-maz@kernel.org> <86msv7ylnu.wl-maz@kernel.org> <05733774-4210-4097-9912-fb3aa8542fdd@oracle.com> <86a5r4zafh.wl-maz@kernel.org> <134912e4-beed-4ab6-8ce1-33e69ec382b3@os.amperecomputing.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gankulkarni@os.amperecomputing.com, miguel.luis@oracle.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, darren@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Fri, 24 Nov 2023 09:50:33 +0000, Ganapatrao Kulkarni wrote: >=20 >=20 >=20 > On 23-11-2023 10:14 pm, Marc Zyngier wrote: > > On Thu, 23 Nov 2023 16:21:48 +0000, > > Miguel Luis wrote: > >>=20 > >> Hi Marc, > >>=20 > >> On 21/11/2023 18:02, Marc Zyngier wrote: > >>> On Tue, 21 Nov 2023 16:49:52 +0000, > >>> Miguel Luis wrote: > >>>> Hi Marc, > >>>>=20 > >>>>> On 20 Nov 2023, at 12:09, Marc Zyngier wrote: > >>>>>=20 > >>>>> This is the 5th drop of NV support on arm64 for this year, and most > >>>>> probably the last one for this side of Christmas. > >>>>>=20 > >>>>> For the previous episodes, see [1]. > >>>>>=20 > >>>>> What's changed: > >>>>>=20 > >>>>> - Drop support for the original FEAT_NV. No existing hardware suppo= rts > >>>>> it without FEAT_NV2, and the architecture is deprecating the form= er > >>>>> entirely. This results in fewer patches, and a slightly simpler > >>>>> model overall. > >>>>>=20 > >>>>> - Reorganise the series to make it a bit more logical now that FEAT= _NV > >>>>> is gone. > >>>>>=20 > >>>>> - Apply the NV idreg restrictions on VM first run rather than on ea= ch > >>>>> access. > >>>>>=20 > >>>>> - Make the nested vgic shadow CPU interface a per-CPU structure rat= her > >>>>> than per-vcpu. > >>>>>=20 > >>>>> - Fix the EL0 timer fastpath > >>>>>=20 > >>>>> - Work around the architecture deficiencies when trapping WFI from a > >>>>> L2 guest. > >>>>>=20 > >>>>> - Fix sampling of nested vgic state (MISR, ELRSR, EISR) > >>>>>=20 > >>>>> - Drop the patches that have already been merged (NV trap forwardin= g, > >>>>> per-MMU VTCR) > >>>>>=20 > >>>>> - Rebased on top of 6.7-rc2 + the FEAT_E2H0 support [2]. > >>>>>=20 > >>>>> The branch containing these patches (and more) is at [3]. As for the > >>>>> previous rounds, my intention is to take a prefix of this series in= to > >>>>> 6.8, provided that it gets enough reviewing. > >>>>>=20 > >>>>> [1] https://lore.kernel.org/r/20230515173103.1017669-1-maz@kernel.o= rg > >>>>> [2] https://lore.kernel.org/r/20231120123721.851738-1-maz@kernel.org > >>>>> [3] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platfor= ms.git/log/?h=3Dkvm-arm64/nv-6.8-nv2-only > >>>>>=20 > >>>> While I was testing this with kvmtool for 5.16 I noted the following= on dmesg: > >>>>=20 > >>>> [ 803.014258] kvm [19040]: Unsupported guest sys_reg access at: 812= 9fa50 [600003c9] > >>>> { Op0( 3), Op1( 5), CRn( 1), CRm( 0), Op2( 2), func= _read }, > >>>>=20 > >>>> This is CPACR_EL12. > >>> CPACR_EL12 is redirected to VNCR[0x100]. It really shouldn't trap... > >>>=20 > >>>> Still need yet to debug. > >>> Can you disassemble the guest around the offending PC? > >>=20 > >> [ 1248.686350] kvm [7013]: Unsupported guest sys_reg access at: 812baa= 50 [600003c9] > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 { Op0( 3), Op1( 5), CRn( 1), CRm( 0), Op2( 2), func_r= ead }, > >>=20 > >> =C2=A012baa00:=C2=A0=C2=A0=C2=A0 14000008 =C2=A0=C2=A0=C2=A0 b=C2=A0= =C2=A0=C2=A0 0x12baa20 > >> =C2=A012baa04:=C2=A0=C2=A0=C2=A0 d000d501 =C2=A0=C2=A0=C2=A0 adrp=C2= =A0=C2=A0=C2=A0 x1, 0x2d5c000 > >> =C2=A012baa08:=C2=A0=C2=A0=C2=A0 91154021 =C2=A0=C2=A0=C2=A0 add=C2= =A0=C2=A0=C2=A0 x1, x1, #0x550 > >> =C2=A012baa0c:=C2=A0=C2=A0=C2=A0 f9400022 =C2=A0=C2=A0=C2=A0 ldr=C2= =A0=C2=A0=C2=A0 x2, [x1] > >> =C2=A012baa10:=C2=A0=C2=A0=C2=A0 f9400421 =C2=A0=C2=A0=C2=A0 ldr=C2= =A0=C2=A0=C2=A0 x1, [x1, #8] > >> =C2=A012baa14:=C2=A0=C2=A0=C2=A0 8a010042 =C2=A0=C2=A0=C2=A0 and=C2= =A0=C2=A0=C2=A0 x2, x2, x1 > >> =C2=A012baa18:=C2=A0=C2=A0=C2=A0 d3441c42 =C2=A0=C2=A0=C2=A0 ubfx=C2= =A0=C2=A0=C2=A0 x2, x2, #4, #4 > >> =C2=A012baa1c:=C2=A0=C2=A0=C2=A0 b4000082 =C2=A0=C2=A0=C2=A0 cbz=C2= =A0=C2=A0=C2=A0 x2, 0x12baa2c > >> =C2=A012baa20:=C2=A0=C2=A0=C2=A0 d2a175a0 =C2=A0=C2=A0=C2=A0 mov=C2= =A0=C2=A0=C2=A0 x0, #0xbad0000=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 // #195887104 > >> =C2=A012baa24:=C2=A0=C2=A0=C2=A0 f2994220 =C2=A0=C2=A0=C2=A0 movk=C2= =A0=C2=A0=C2=A0 x0, #0xca11 > >> =C2=A012baa28:=C2=A0=C2=A0=C2=A0 d69f03e0 =C2=A0=C2=A0=C2=A0 eret > >> =C2=A012baa2c:=C2=A0=C2=A0=C2=A0 d2c00080 =C2=A0=C2=A0=C2=A0 mov=C2= =A0=C2=A0=C2=A0 x0, #0x400000000=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 // #17179869184 > >> =C2=A012baa30:=C2=A0=C2=A0=C2=A0 f2b10000 =C2=A0=C2=A0=C2=A0 movk=C2= =A0=C2=A0=C2=A0 x0, #0x8800, lsl #16 > >> =C2=A012baa34:=C2=A0=C2=A0=C2=A0 f2800000 =C2=A0=C2=A0=C2=A0 movk=C2= =A0=C2=A0=C2=A0 x0, #0x0 > >> =C2=A012baa38:=C2=A0=C2=A0=C2=A0 d51c1100 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 hcr_el2, x0 > >> =C2=A012baa3c:=C2=A0=C2=A0=C2=A0 d5033fdf =C2=A0=C2=A0=C2=A0 isb > >> =C2=A012baa40:=C2=A0=C2=A0=C2=A0 d53c4100 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, sp_el1 > >> =C2=A012baa44:=C2=A0=C2=A0=C2=A0 9100001f =C2=A0=C2=A0=C2=A0 mov=C2= =A0=C2=A0=C2=A0 sp, x0 > >> =C2=A012baa48:=C2=A0=C2=A0=C2=A0 d538d080 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, tpidr_el1 > >> =C2=A012baa4c:=C2=A0=C2=A0=C2=A0 d51cd040 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 tpidr_el2, x0 > >> =C2=A012baa50:=C2=A0=C2=A0=C2=A0 d53d1040 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, cpacr_el12 > >> =C2=A012baa54:=C2=A0=C2=A0=C2=A0 d5181040 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 cpacr_el1, x0 > >> =C2=A012baa58:=C2=A0=C2=A0=C2=A0 d53dc000 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, vbar_el12 > >> =C2=A012baa5c:=C2=A0=C2=A0=C2=A0 d518c000 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 vbar_el1, x0 > >> =C2=A012baa60:=C2=A0=C2=A0=C2=A0 d53c1120 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, mdcr_el2 > >> =C2=A012baa64:=C2=A0=C2=A0=C2=A0 9272f400 =C2=A0=C2=A0=C2=A0 and=C2= =A0=C2=A0=C2=A0 x0, x0, #0xffffffffffffcfff > >> =C2=A012baa68:=C2=A0=C2=A0=C2=A0 9266f400 =C2=A0=C2=A0=C2=A0 and=C2= =A0=C2=A0=C2=A0 x0, x0, #0xfffffffffcffffff > >> =C2=A012baa6c:=C2=A0=C2=A0=C2=A0 d51c1120 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 mdcr_el2, x0 > >> =C2=A012baa70:=C2=A0=C2=A0=C2=A0 d53d2040 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, tcr_el12 > >> =C2=A012baa74:=C2=A0=C2=A0=C2=A0 d5182040 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 tcr_el1, x0 > >> =C2=A012baa78:=C2=A0=C2=A0=C2=A0 d53d2000 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, ttbr0_el12 > >> =C2=A012baa7c:=C2=A0=C2=A0=C2=A0 d5182000 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 ttbr0_el1, x0 > >> =C2=A012baa80:=C2=A0=C2=A0=C2=A0 d53d2020 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, ttbr1_el12 > >> =C2=A012baa84:=C2=A0=C2=A0=C2=A0 d5182020 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 ttbr1_el1, x0 > >> =C2=A012baa88:=C2=A0=C2=A0=C2=A0 d53da200 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, mair_el12 > >> =C2=A012baa8c:=C2=A0=C2=A0=C2=A0 d518a200 =C2=A0=C2=A0=C2=A0 msr=C2= =A0=C2=A0=C2=A0 mair_el1, x0 > >> =C2=A012baa90:=C2=A0=C2=A0=C2=A0 d5380761 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x1, s3_0_c0_c7_3 > >> =C2=A012baa94:=C2=A0=C2=A0=C2=A0 d3400c21 =C2=A0=C2=A0=C2=A0 ubfx=C2= =A0=C2=A0=C2=A0 x1, x1, #0, #4 > >> =C2=A012baa98:=C2=A0=C2=A0=C2=A0 b4000141 =C2=A0=C2=A0=C2=A0 cbz=C2= =A0=C2=A0=C2=A0 x1, 0x12baac0 > >> =C2=A012baa9c:=C2=A0=C2=A0=C2=A0 d53d2060 =C2=A0=C2=A0=C2=A0 mrs=C2= =A0=C2=A0=C2=A0 x0, s3_5_c2_c0_3 > >=20 > > OK, this is suspiciously close to the location Ganapatrao was having > > issues with. Are you running on the same hardware? > >=20 > > In any case, we should never take a trap for this access. Can you dump > > HCR_EL2 at the point where the guest traps (in switch.c)? > >=20 >=20 > I have dumped HCR_EL2 before entry to L1 in both V11 and V10. > on V10 HCR_EL2=3D0x2743c827c263f > on V11 HCR_EL2=3D0x27c3c827c263f >=20 > on V11 the function vcpu_el2_e2h_is_set(vcpu) is returning false > resulting in NV1 bit set along with NV and NV2. > AFAIK, For L1 to be in VHE, NV1 bit should be zero and NV=3DNV2=3D1. >=20 > I could boot L1 then L2, if I hack vcpu_el2_e2h_is_set to return true. > There could be a bug in V11 or E2H0 patchset resulting in > vcpu_el2_e2h_is_set() returning false? The E2H0 series should only force vcpu_el2_e2h_is_set() to return true, but not set it to false. Can you dump the *guest's* version of HCR_EL2 at this point? M. --=20 Without deviation from the norm, progress is not possible.