From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0226EC7EE24 for ; Tue, 6 Jun 2023 09:34:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vGms2X+J9I18ZuO3rB2ucTtEXIH3WDQoymI5eaSI5XA=; b=ZTTAW416y9EN/l 7xPhK0hUJDrEAxwa+JXX2h0kv5aB5RlQPyBJcqcpgETv6p9dhOamspPZMJ2iMvsIoLgT4EdAciYtv vRH4qA96cr0yBpYQtsmqTkwg05Qn13PZleG6oUZ4GyD2pjD+QwpcqESjxCMeHOayZuhLSuM7FMZ1C siUqc31VJhnSUUGURYCAX2QWD8zIGPtAVvgHR1xp00o5m2BDPkLuS4jBoSCaSwTp09q7APWy6Ubx+ fZCajC7NNjW5ygTC9ohxGnRLTJRzi701FlJGTiF6lwjbN2HD5zqzVAgrmUhw7S+xvWxeAhYQKYeNF wCbgDj9LDni9ogp8vCig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q6T4N-0010v0-38; Tue, 06 Jun 2023 09:33:39 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q6T4K-0010tn-0q for linux-arm-kernel@lists.infradead.org; Tue, 06 Jun 2023 09:33:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686044015; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NfCVlNZr5FMM7IYZiy0Efi0L7IH3DzTFtcUbvK87ioM=; b=V82Qxj3GuefY+sFBeLR5sV9ksAMnTHEKcZGFbARwiNHejCv1F9Q9vSxq73Ns3u25Fsj2Mx ecanYjldmOwcm3oBDOTtGoQ7FAoh3eYwJ1PQ1+HqffCubUVt1F5ZhNfj/A+Hn2AT9+qTZS uZHr/cfWHZR6WzOzoYOeK94Sgis5yUM= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-650-m0NlNWlJP1C8kTplhblOqA-1; Tue, 06 Jun 2023 05:33:34 -0400 X-MC-Unique: m0NlNWlJP1C8kTplhblOqA-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-3f7f713eeb0so89263981cf.1 for ; Tue, 06 Jun 2023 02:33:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686044012; x=1688636012; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NfCVlNZr5FMM7IYZiy0Efi0L7IH3DzTFtcUbvK87ioM=; b=Vo1m+ze9Gdf/8euYuPxcS/IBjZCmu7f60+fJ2krUFMfw+M5kSl035oCR617/fHDldz sPiBvulKxWtRUTmd8rCnesa54stmRx9Dv4kgjmqr/HkTt+Eqwcnz/ob+0Tgqub3bruo0 NsfBBexWCUHFd+nwFtQCCbBBCUt6dkAZR8aX0z0c8MZg2Vem5jZrh4fJjGjUwgNzxEnV +dMhxUR5Qom7/u/M9CMgz9VLvuQGwb15gb7GuIP/6k/ZTPmF3yDT9jGypooBKHovnrmn uR9oqLbvS/xgoU+pcxr/fBqhpwyPtQ9NVLVrTYABKe/DR2yTV2kKoVd4K0Xjp27CHsgJ YF3Q== X-Gm-Message-State: AC+VfDwnjjpks7UfwLK1C7rlelFf5ZcvxIy7zz5HzXu8ids0iMkdSpY3 1neNuvhDkBtXAH8o0gX72iyi72P7vPpny2hqnd4dTLAXFhKRbtfMUdconAYHcyGQ6hDCo5OOJOR qLVyL4YSJLMl07SXJLvdvvId1BAg/8L6pgdeFBdpSCww= X-Received: by 2002:ac8:7c54:0:b0:3f5:1626:6a3d with SMTP id o20-20020ac87c54000000b003f516266a3dmr1313722qtv.42.1686044012688; Tue, 06 Jun 2023 02:33:32 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5n5fSaR9rhGQJeywpdZviphMffC6+N0hasH8JtAg/z6WKXKQqm2K/P4sycA2R8M1LOqxyk5Q== X-Received: by 2002:ac8:7c54:0:b0:3f5:1626:6a3d with SMTP id o20-20020ac87c54000000b003f516266a3dmr1313701qtv.42.1686044012476; Tue, 06 Jun 2023 02:33:32 -0700 (PDT) Received: from ?IPV6:2a01:e0a:59e:9d80:527b:9dff:feef:3874? ([2a01:e0a:59e:9d80:527b:9dff:feef:3874]) by smtp.gmail.com with ESMTPSA id r12-20020ac85e8c000000b003f9ad6acba4sm634844qtx.79.2023.06.06.02.33.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 06 Jun 2023 02:33:31 -0700 (PDT) Message-ID: Date: Tue, 6 Jun 2023 11:33:27 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.5.0 Subject: Re: [PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support To: Marc Zyngier Cc: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Ganapatrao Kulkarni , Darren Hart , Jintack Lim , Russell King , Miguel Luis , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu References: <20230515173103.1017669-1-maz@kernel.org> <16d9fda4-3ead-7d5e-9f54-ef29fbd932ac@redhat.com> <87zg64nhqh.wl-maz@kernel.org> <86r0rfkpwd.wl-maz@kernel.org> From: Eric Auger In-Reply-To: <86r0rfkpwd.wl-maz@kernel.org> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230606_023336_368582_DEE0AB9F X-CRM114-Status: GOOD ( 26.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Marc, On 5/17/23 16:12, Marc Zyngier wrote: > On Wed, 17 May 2023 09:59:45 +0100, > Eric Auger wrote: >> >> Hi Marc, >>Hi Marc, >> On 5/16/23 22:28, Marc Zyngier wrote: >>> On Tue, 16 May 2023 17:53:14 +0100, >>> Eric Auger wrote: >>>> >>>> Hi Marc, >>>> >>>> On 5/15/23 19:30, Marc Zyngier wrote: >>>>> This is the 4th drop of NV support on arm64 for this year. >>>>> >>>>> For the previous episodes, see [1]. >>>>> >>>>> What's changed: >>>>> >>>>> - New framework to track system register traps that are reinjected in >>>>> guest EL2. It is expected to replace the discrete handling we have >>>>> enjoyed so far, which didn't scale at all. This has already fixed a >>>>> number of bugs that were hidden (a bunch of traps were never >>>>> forwarded...). Still a work in progress, but this is going in the >>>>> right direction. >>>>> >>>>> - Allow the L1 hypervisor to have a S2 that has an input larger than >>>>> the L0 IPA space. This fixes a number of subtle issues, depending on >>>>> how the initial guest was created. >>>>> >>>>> - Consequently, the patch series has gone longer again. Boo. But >>>>> hopefully some of it is easier to review... >>>>> >>>>> [1] https://lore.kernel.org/r/20230405154008.3552854-1-maz@kernel.org >>>> >>>> I have started testing this and when booting my fedora guest I get >>>> >>>> [ 151.796544] kvm [7617]: Unsupported guest sys_reg access at: >>>> 23f425fd0 [80000209] >>>> [ 151.796544] { Op0( 3), Op1( 3), CRn(14), CRm( 3), Op2( 1), func_write }, >>>> >>>> as soon as the host has kvm-arm.mode=nested >>>> >>>> This seems to be triggered very early by EDK2 >>>> (ArmPkg/Drivers/TimerDxe/TimerDxe.c). >>>> >>>> If I am not wrong this CNTV_CTL_EL0. Do you have any idea? >>> >>> So here's my current analysis: >>> >>> I assume you are running EDK2 as the L1 guest in a nested >>> configuration. I also assume that you are not running on an Apple >>> CPU. If these assumptions are correct, then EDK2 runs at vEL2, and is >>> in nVHE mode. >>> >>> Finally, I'm going to assume that your implementation has FEAT_ECV and >>> FEAT_NV2, because I can't see how it could fail otherwise. >> all the above is correct. >>> >>> In these precise conditions, KVM sets the CNTHCTL_EL2.EL1TVT bit so >>> that we can trap the EL0 virtual timer and faithfully emulate it (it >>> is otherwise written to memory, which isn't very helpful). >> >> indeed >>> >>> As it turns out, we don't handle these traps. I didn't spot it because >>> my test machines are all Apple boxes that don't have a nVHE mode, so >>> nothing on the nVHE path is getting *ANY* coverage. Hint: having >>> access to such a machine would help (shipping address on request!). >>> Otherwise, I'll eventually kill the nVHE support altogether. >>> >>> I have written the following patch, which compiles, but that I cannot >>> test with my current setup. Could you please give it a go? >> >> with the patch below, my guest boots nicely. You did it great on the 1st >> shot!!! So this fixes my issue. I will continue testing the v10. > > Thanks a lot for reporting the issue and testing my hacks. I'll > eventually fold it into the rest of the series. > > By the way, what are you using as your VMM? I'd really like to > reproduce your setup. Sorry I missed your reply. I am using libvirt + qemu (feat Miguel's RFC) and fedora L1 guest. Thanks to your fix, this boots fine. But at the moment it does not reboot and hangs in edk2 I think. Unfortunately this time I have no trace on host :-( While looking at your series I will add some traces. Eric > > Cheers, > > M. > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel