From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78DB4C64E7A for ; Tue, 1 Dec 2020 13:47:42 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EC6F020770 for ; Tue, 1 Dec 2020 13:47:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="mkjGO6sk"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="OAGcRA/1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EC6F020770 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=u+lejOuMgYndscVRKXEEf2OpUUTxWeBcWZeOrFpxrno=; b=mkjGO6sk5QktMavBSSHJe1joG Y0TpZQR3M+IV+/meJdb0O8lCQGOu8RaFRE+HTGj0m8Vw9PlNi1bJluN+mW2fe/dlJcXUdlJDYTBaE QU5L27ncJ3EfQsWFP664DyTq+rm1vlvDi+zgGHbZZo7i36KcdsQBacNG9UI415o1q8VFSTeIwvEem T25MpzwSbJfhcTKZbSRFhvFfkhy6cM3RfmcSDtEF9pkkit+Yl8ch7z5lRGvmuVNq+HZBQS5YlmCmB /i/w5EXajTQrgOujf+3349XqYcy0n9PCnfU6p57Gh9XxydQsk1M+gDI3wB6lMPlb8tFzylHkTT7C6 CqZo+0MtQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kk5zA-0008Iz-FJ; Tue, 01 Dec 2020 13:46:28 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kk5yw-0008Dh-Pp for linux-arm-kernel@lists.infradead.org; Tue, 01 Dec 2020 13:46:24 +0000 Received: from willie-the-truck (236.31.169.217.in-addr.arpa [217.169.31.236]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A0FC720770; Tue, 1 Dec 2020 13:46:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1606830373; bh=jKd8KoPjJ69TJ40PmJD4Wfy02BVwL1rHNuh/7ARwmdw=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=OAGcRA/1u/yCOlF1mmmsp9BdD5Y5uHULGMAke3nSBKdj04HTbZkQmetJT8zFYNvna D/ZoIqBmzjQognFPlcRnXsrnouuI4dmUbY/yqAihwcs1ySMvyTpvYsRTcIiCbKTdqy ujr4XK9GDSeDgqokeQMdRxMeDdAqbl/coBb7ZajE= Date: Tue, 1 Dec 2020 13:46:07 +0000 From: Will Deacon To: "wangyanan (Y)" Subject: Re: [RFC PATCH 2/3] KVM: arm64: Fix handling of merging tables into a block entry Message-ID: <20201201134606.GB26973@willie-the-truck> References: <20201130121847.91808-1-wangyanan55@huawei.com> <20201130121847.91808-3-wangyanan55@huawei.com> <20201130133421.GB24837@willie-the-truck> <67e9e393-1836-eca7-4235-6f4a19fed652@huawei.com> <20201130160119.GA25051@willie-the-truck> <868a4403-10d3-80f3-4ae1-a490813c55e2@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <868a4403-10d3-80f3-4ae1-a490813c55e2@huawei.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201201_084615_069393_B5FBB22A X-CRM114-Status: GOOD ( 39.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: jiangkunkun@huawei.com, Gavin Shan , Suzuki K Poulose , Marc Zyngier , wangjingyi11@huawei.com, Quentin Perret , lushenming@huawei.com, linux-kernel@vger.kernel.org, yezengruan@huawei.com, James Morse , linux-arm-kernel@lists.infradead.org, Catalin Marinas , yuzenghui@huawei.com, wanghaibin.wang@huawei.com, zhukeqian1@huawei.com, Julien Thierry Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Dec 01, 2020 at 10:30:41AM +0800, wangyanan (Y) wrote: > On 2020/12/1 0:01, Will Deacon wrote: > > On Mon, Nov 30, 2020 at 11:24:19PM +0800, wangyanan (Y) wrote: > > > On 2020/11/30 21:34, Will Deacon wrote: > > > > On Mon, Nov 30, 2020 at 08:18:46PM +0800, Yanan Wang wrote: > > > > > diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pg= table.c > > > > > index 696b6aa83faf..fec8dc9f2baa 100644 > > > > > --- a/arch/arm64/kvm/hyp/pgtable.c > > > > > +++ b/arch/arm64/kvm/hyp/pgtable.c > > > > > @@ -500,6 +500,9 @@ static int stage2_map_walk_table_pre(u64 addr= , u64 end, u32 level, > > > > > return 0; > > > > > } > > > > > +static void stage2_flush_dcache(void *addr, u64 size); > > > > > +static bool stage2_pte_cacheable(kvm_pte_t pte); > > > > > + > > > > > static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, = kvm_pte_t *ptep, > > > > > struct stage2_map_data *data) > > > > > { > > > > > @@ -507,9 +510,17 @@ static int stage2_map_walk_leaf(u64 addr, u6= 4 end, u32 level, kvm_pte_t *ptep, > > > > > struct page *page =3D virt_to_page(ptep); > > > > > if (data->anchor) { > > > > > - if (kvm_pte_valid(pte)) > > > > > + if (kvm_pte_valid(pte)) { > > > > > + kvm_set_invalid_pte(ptep); > > > > > + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, > > > > > + addr, level); > > > > > put_page(page); > > > > This doesn't make sense to me: the page-table pages we're walking w= hen the > > > > anchor is set are not accessible to the hardware walker because we = unhooked > > > > the entire sub-table in stage2_map_walk_table_pre(), which has the = necessary > > > > TLB invalidation. > > > > = > > > > Are you seeing a problem in practice here? > > > Yes, I indeed find a problem in practice. > > > = > > > When the migration was cancelled, a TLB conflic abort=A0 was found in= guest. > > > = > > > This problem is fixed before rework of the page table code, you can h= ave a > > > look in the following two links: > > > = > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/co= mmit/?id=3D3c3736cd32bf5197aed1410ae826d2d254a5b277 > > > = > > > https://lists.cs.columbia.edu/pipermail/kvmarm/2019-March/035031.html > > Ok, let's go through this, because I still don't see the bug. Please co= rrect > > me if you spot any mistakes: > > = > > 1. We have a block mapping for X =3D> Y > > 2. Dirty logging is enabled, so the block mapping is write-protected= and > > ends up being split into page mappings > > 3. Dirty logging is disabled due to a failed migration. > > = > > --- At this point, I think we agree that the state of the MMU is alrigh= t --- > > = > > 4. We take a stage-2 fault and want to reinstall the block mapping: > > = > > a. kvm_pgtable_stage2_map() is invoked to install the block mappi= ng > > b. stage2_map_walk_table_pre() finds a table where we would like = to > > install the block: > > = > > i. The anchor is set to point at this entry > > ii. The entry is made invalid > > iii. We invalidate the TLB for the input address. This is > > TLBI IPAS2SE1IS without level hint and then TLBI VMALLE1IS. > > = > > *** At this point, the page-table pointed to by the old table entry > > is not reachable to the hardware walker *** > > = > > c. stage2_map_walk_leaf() is called for each leaf entry in the > > now-unreachable subtree, dropping page-references for each val= id > > entry it finds. > > d. stage2_map_walk_table_post() is eventually called for the entry > > which we cleared back in b.ii, so we install the new block map= ping. > > = > > You are proposing to add additional TLB invalidation to (c), but I don't > > think that is necessary, thanks to the invalidation already performed in > > b.iii. What am I missing here? > = > The point is at b.iii where the TLBI is not enough. There are many page > mappings that we need to merge into a block mapping. > = > We invalidate the TLB for the input address without level hint at b.iii, = but > this operation just flush TLB for one page mapping, there > = > are still some TLB entries for the other page mappings in the cache, the = MMU > hardware walker can still hit these entries next time. Ah, yes, I see. Thanks. I hadn't considered the case where there are table entries beneath the anchor. So how about the diff below? Will --->8 diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 0271b4a3b9fe..12526d8c7ae4 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -493,7 +493,7 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end,= u32 level, return 0; = kvm_set_invalid_pte(ptep); - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, 0); + /* TLB invalidation is deferred until the _post handler */ data->anchor =3D ptep; return 0; } @@ -547,11 +547,21 @@ static int stage2_map_walk_table_post(u64 addr, u64 e= nd, u32 level, struct stage2_map_data *data) { int ret =3D 0; + kvm_pte_t pte =3D *ptep; = if (!data->anchor) return 0; = - free_page((unsigned long)kvm_pte_follow(*ptep)); + kvm_set_invalid_pte(ptep); + + /* + * Invalidate the whole stage-2, as we may have numerous leaf + * entries below us which would otherwise need invalidating + * individually. + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); + + free_page((unsigned long)kvm_pte_follow(pte)); put_page(virt_to_page(ptep)); = if (data->anchor =3D=3D ptep) { _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel