From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96C6AC77B75 for ; Tue, 16 May 2023 18:47:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229635AbjEPSrT (ORCPT ); Tue, 16 May 2023 14:47:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229574AbjEPSrR (ORCPT ); Tue, 16 May 2023 14:47:17 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6FE483A87 for ; Tue, 16 May 2023 11:47:04 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E24E062CD7 for ; Tue, 16 May 2023 18:47:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3C68FC433EF; Tue, 16 May 2023 18:47:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1684262823; bh=Ai5dTWFXnWb/13sR7qeK7xi7T0l15zxYtlSLz/ku5wA=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=Hz3PPIDO1FiENcBWZQToZ27EN+jGg4NNFnKm0ajFY4LCEQ1NblgutI9xXMdwJmI0R 36VZEg8w3Y/XvVJpxI51ZWbwed4xiNIY2RUww8+nGZbYUW+PQCP1cz3+CqOQ0UB3Yf 8+pMJGWZ2NQkr8tMmdctIBGp4ctHqh00YnhpmQcXeLFI/xRih0Si0grhtLkSDG1T3S Z9hrTZ9e5VU7s7jX42VqPKGDgqWLrCG78c9GEQAzf38MoBOBgMkUO1CeG8ChOb9XbQ onD6UHBan0zQf0k7MmTEYkZMxXWDT8IwkhaOBiaH4YnjcqSClreUFdB89a5Lw3QapA p6nzof97QaQVA== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pyzhM-00FdjW-Sq; Tue, 16 May 2023 19:47:01 +0100 Date: Tue, 16 May 2023 19:47:00 +0100 Message-ID: <86y1loktaz.wl-maz@kernel.org> From: Marc Zyngier To: Eric Auger Cc: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Darren Hart , Jintack Lim , Russell King , Miguel Luis , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: Re: [PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support In-Reply-To: <16d9fda4-3ead-7d5e-9f54-ef29fbd932ac@redhat.com> References: <20230515173103.1017669-1-maz@kernel.org> <16d9fda4-3ead-7d5e-9f54-ef29fbd932ac@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: eauger@redhat.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, gankulkarni@os.amperecomputing.com, darren@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, miguel.luis@oracle.com, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Hey Eric, On Tue, 16 May 2023 17:53:14 +0100, Eric Auger wrote: > > Hi Marc, > > On 5/15/23 19:30, Marc Zyngier wrote: > > This is the 4th drop of NV support on arm64 for this year. > > > > For the previous episodes, see [1]. > > > > What's changed: > > > > - New framework to track system register traps that are reinjected in > > guest EL2. It is expected to replace the discrete handling we have > > enjoyed so far, which didn't scale at all. This has already fixed a > > number of bugs that were hidden (a bunch of traps were never > > forwarded...). Still a work in progress, but this is going in the > > right direction. > > > > - Allow the L1 hypervisor to have a S2 that has an input larger than > > the L0 IPA space. This fixes a number of subtle issues, depending on > > how the initial guest was created. > > > > - Consequently, the patch series has gone longer again. Boo. But > > hopefully some of it is easier to review... > > > > [1] https://lore.kernel.org/r/20230405154008.3552854-1-maz@kernel.org > > I have started testing this and when booting my fedora guest I get > > [ 151.796544] kvm [7617]: Unsupported guest sys_reg access at: > 23f425fd0 [80000209] > [ 151.796544] { Op0( 3), Op1( 3), CRn(14), CRm( 3), Op2( 1), func_write }, > > as soon as the host has kvm-arm.mode=nested Very odd. A write to CNTV_CTL_EL0 that fails, meaning that we have ECV traps for the virtual timer set, and so this is probably done from (virtual) EL2. Which of course we're not properly handling. Duh. > This seems to be triggered very early by EDK2 > (ArmPkg/Drivers/TimerDxe/TimerDxe.c). > > If I am not wrong this CNTV_CTL_EL0. Do you have any idea? I have a good idea, but I could use some more information: - Is this a nested guest? Or a non nested guest? - Can you stash your EDK2 build somewhere so that I can try to reproduce it? - What do you use for VMM on the host? I'm running EDK2 on both L1 and L2 guests (some Debian builds), and I don't see any issue. But when running EDK2 on L1, I run it non-nested. > By the way I got this already with your v9 > (git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git > kvm-arm64/nv-6.4-WIP) and with your v10 (I cherry-picked your > maz/kvm-arm64/nv-6.5-WIP branch). That's an interesting data point. At least, it hasn't regressed further... I'll try and cook a test tomorrow and run it on my own rig. Which may or may not behave like yours, you never know... Thanks a lot for the report! M. -- Without deviation from the norm, progress is not possible.