From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0096C2D0F7 for ; Tue, 12 May 2020 16:13:46 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 778C8206B7 for ; Tue, 12 May 2020 16:13:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="tzNdeFw4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 778C8206B7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Content-ID:Content-Description :Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OxNi2AP84vj4B8fmC+uuayWOLrl6e9lSfEollGJp4Ps=; b=tzNdeFw4gO7PR6 1gxQWQydva6hO3UccqKW+7u4sjB6YRTYO+ALqmmRKpaLrIBz3sZjrrl/u7cx5WyDif+PZgi/2I2T1 x3fLpWfKqtTmKgiVeX7OL7nIebgD14EWbG8pbxNy3OWaKTWjMiuqgnO4d72kAlkdDZPtCwr2t286P BNWy/xEmA8/RVFZ2wlGQQJo85GVnWTf9CrBusRQE3FjtkPHQlP4MiJKadZesFDqS6eKDgtfK2bfiv hKoDLqNYLuDmhgZXsUBagu8psMd1gL1AtpIx/OYwT4a67LCegJ6rnfb8Z1+rL83vzTO92qXN6gESL cfzjP1433sY95C3oDr9w==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYXXN-0003sp-SM; Tue, 12 May 2020 16:13:45 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYXXI-0003nX-CN for linux-arm-kernel@lists.infradead.org; Tue, 12 May 2020 16:13:42 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 595BC1FB; Tue, 12 May 2020 09:13:39 -0700 (PDT) Received: from [192.168.0.14] (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 09D253F305; Tue, 12 May 2020 09:13:36 -0700 (PDT) Subject: Re: [PATCH 03/26] KVM: arm64: Factor out stage 2 page table data from struct kvm To: Alexandru Elisei References: <20200422120050.3693593-1-maz@kernel.org> <20200422120050.3693593-4-maz@kernel.org> <76d811eb-b304-c49f-1f21-fe9d95112a28@arm.com> <5134e123-18ec-9b69-2e0a-b83798e01507@arm.com> From: James Morse Message-ID: Date: Tue, 12 May 2020 17:13:35 +0100 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <5134e123-18ec-9b69-2e0a-b83798e01507@arm.com> Content-Language: en-GB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200512_091340_511384_5904D2E0 X-CRM114-Status: GOOD ( 25.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Catalin Marinas , kvm@vger.kernel.org, Suzuki K Poulose , Jintack Lim , Marc Zyngier , Christoffer Dall , Dave Martin , George Cherian , Julien Thierry , "Zengtao \(B\)" , Andre Przywara , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Alex, On 12/05/2020 16:47, Alexandru Elisei wrote: > On 5/12/20 12:17 PM, James Morse wrote: >> On 11/05/2020 17:38, Alexandru Elisei wrote: >>> On 4/22/20 1:00 PM, Marc Zyngier wrote: >>>> From: Christoffer Dall >>>> >>>> As we are about to reuse our stage 2 page table manipulation code for >>>> shadow stage 2 page tables in the context of nested virtualization, we >>>> are going to manage multiple stage 2 page tables for a single VM. >>>> >>>> This requires some pretty invasive changes to our data structures, >>>> which moves the vmid and pgd pointers into a separate structure and >>>> change pretty much all of our mmu code to operate on this structure >>>> instead. >>>> >>>> The new structure is called struct kvm_s2_mmu. >>>> >>>> There is no intended functional change by this patch alone. >>>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h >>>> index 7dd8fefa6aecd..664a5d92ae9b8 100644 >>>> --- a/arch/arm64/include/asm/kvm_host.h >>>> +++ b/arch/arm64/include/asm/kvm_host.h >>>> @@ -63,19 +63,32 @@ struct kvm_vmid { >>>> u32 vmid; >>>> }; >>>> >>>> -struct kvm_arch { >>>> +struct kvm_s2_mmu { >>>> struct kvm_vmid vmid; >>>> >>>> - /* stage2 entry level table */ >>>> - pgd_t *pgd; >>>> - phys_addr_t pgd_phys; >>>> - >>>> - /* VTCR_EL2 value for this VM */ >>>> - u64 vtcr; >>>> + /* >>>> + * stage2 entry level table >>>> + * >>>> + * Two kvm_s2_mmu structures in the same VM can point to the same pgd >>>> + * here. This happens when running a non-VHE guest hypervisor which >>>> + * uses the canonical stage 2 page table for both vEL2 and for vEL1/0 >>>> + * with vHCR_EL2.VM == 0. >>> It makes more sense to me to say that a non-VHE guest hypervisor will use the >>> canonical stage *1* page table when running at EL2 >> Can KVM say anything about stage1? Its totally under the the guests control even at vEL2... > It is. My interpretation of the comment was that if the guest doesn't have virtual > stage 2 enabled (we're not running a guest of the L1 hypervisor), then the L0 host > can use the same L0 stage 2 tables because we're running the same guest (the L1 > VM), regardless of the actual exception level for the guest. I think you're right, but I can't see where stage 1 comes in to it! > If I remember > correctly, KVM assigns different vmids for guests running at vEL1/0 and vEL2 with > vHCR_EL2.VM == 0 because the translation regimes are different, but keeps the same > translation tables. Interesting. Is that because vEL2 really has ASIDs so it needs its own VMID space? >>> (the "Non-secure EL2 translation regime" as ARM DDI 0487F.b calls it on page D5-2543). >>> I think that's >>> the only situation where vEL2 and vEL1&0 will use the same L0 stage 2 tables. It's >>> been quite some time since I reviewed the initial version of the NV patches, did I >>> get that wrong? >> >>>> + */ >>>> + pgd_t *pgd; >>>> + phys_addr_t pgd_phys; >>>> >>>> /* The last vcpu id that ran on each physical CPU */ >>>> int __percpu *last_vcpu_ran; >>> It makes sense for the other fields to be part of kvm_s2_mmu, but I'm struggling >>> to figure out why last_vcpu_ran is here. Would you mind sharing the rationale? I >>> don't see this change in v1 or v2 of the NV series. >> Marc may have a better rationale. My thinking was because kvm_vmid is in here too. >> >> last_vcpu_ran exists to prevent KVM accidentally emulating CNP without the opt-in. (we >> call it defacto CNP). >> >> The guest may expect to be able to use asid-4 with different page tables on different > I'm afraid I don't know what asid-4 is. Sorry - 4 was just a random number![0] 'to use the same asid number on different vcpus'. >> vCPUs, assuming the TLB isn't shared. But if KVM is switching between those vCPU on one >> physical CPU, the TLB is shared, ... the VMID and ASID are the same, but the page tables >> are not. Not fun to debug! >> >> >> NV makes this problem per-stage2, because each stage2 has its own VMID, we need to track >> the vcpu_id that last ran this stage2 on this physical CPU. If its not the same, we need >> to blow away this VMIDs TLB entries. >> >> The workaround lives in virt/kvm/arm/arm.c::kvm_arch_vcpu_load() > > Makes sense, thank you for explaining that. Great, Thanks, James [0] https://xkcd.com/221/ _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel