From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2D9C241FC for ; Tue, 26 Sep 2023 15:20:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 117E9C433C8; Tue, 26 Sep 2023 15:20:05 +0000 (UTC) Date: Tue, 26 Sep 2023 16:20:03 +0100 From: Catalin Marinas To: Shameerali Kolothum Thodi Cc: Oliver Upton , "kvmarm@lists.linux.dev" , "kvm@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "maz@kernel.org" , "will@kernel.org" , "james.morse@arm.com" , "suzuki.poulose@arm.com" , yuzenghui , zhukeqian , Jonathan Cameron , Linuxarm Subject: Re: [RFC PATCH v2 3/8] KVM: arm64: Add some HW_DBM related pgtable interfaces Message-ID: References: <20230825093528.1637-1-shameerali.kolothum.thodi@huawei.com> <20230825093528.1637-4-shameerali.kolothum.thodi@huawei.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Mon, Sep 25, 2023 at 08:04:39AM +0000, Shameerali Kolothum Thodi wrote: > From: Oliver Upton [mailto:oliver.upton@linux.dev] > > On Fri, Sep 22, 2023 at 04:24:11PM +0100, Catalin Marinas wrote: > > > I was wondering if this interferes with the OS dirty tracking (not the > > > KVM one) but I think that's ok, at least at this point, since the PTE is > > > already writeable and a fault would have marked the underlying page as > > > dirty (user_mem_abort() -> kvm_set_pfn_dirty()). > > > > > > I'm not particularly fond of relying on this but I need to see how it > > > fits with the rest of the series. IIRC KVM doesn't go around and make > > > Stage 2 PTEs read-only but rather unmaps them when it changes the > > > permission of the corresponding Stage 1 VMM mapping. > > > > > > My personal preference would be to track dirty/clean properly as we do > > > for stage 1 (e.g. DBM means writeable PTE) but it has some downsides > > > like the try_to_unmap() code having to retrieve the dirty state via > > > notifiers. > > > > KVM's usage of DBM is complicated by the fact that the dirty log > > interface w/ userspace is at PTE granularity. We only want the page > > table walker to relax PTEs, but take faults on hugepages so we can do > > page splitting. Thanks for the clarification. > > > > @@ -952,6 +990,11 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx, > > > > stage2_pte_executable(new)) > > > > mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); > > > > > > > > + /* Save the possible hardware dirty info */ > > > > + if ((ctx->level == KVM_PGTABLE_MAX_LEVELS - 1) && > > > > + stage2_pte_writeable(ctx->old)) > > > > + mark_page_dirty(kvm_s2_mmu_to_kvm(pgt->mmu), ctx->addr >> PAGE_SHIFT); > > > > + > > > > stage2_make_pte(ctx, new); > > > > > > Isn't this racy and potentially losing the dirty state? Or is the 'new' > > > value guaranteed to have the S2AP[1] bit? For stage 1 we normally make > > > the page genuinely read-only (clearing DBM) in a cmpxchg loop to > > > preserve the dirty state (see ptep_set_wrprotect()). > > > > stage2_try_break_pte() a few lines up does a cmpxchg() and full > > break-before-make, so at this point there shouldn't be a race with > > either software or hardware table walkers. Ah, I missed this. Also it was unrelated to this patch (or rather not introduced by this patch). > > In both cases the 'old' translation should have DBM cleared. Even if the > > PTE were dirty, this is wasted work since we need to do a final scan of > > the stage-2 when userspace collects the dirty log. > > > > Am I missing something? > > I think we can get rid of the above mark_page_dirty(). I will test it to confirm > we are not missing anything here. Is this the case for the other places of mark_page_dirty() in your patches? If stage2_pte_writeable() is true, it must have been made writeable earlier by a fault and the underlying page marked as dirty. -- Catalin From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C5245E7E644 for ; Tue, 26 Sep 2023 15:20:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7qDxOLvxA7l5J7xtNEWxKWdIT7m0wH2nwhfFWB6L70o=; b=XPaQtTXG231Hdf nYnq7TU6HO0oDRPjFQqG0RRwYkMbnxox0zxObMyPdpg6QQpDbv/fQrd7mGlYrCxMyVjsntnaEQS08 Hoym3akrg55uS5BGUK/1ITVXmpPyQRekcc7r8PeVuSJbXXUFZ46Q1otMyVyGNzQNOoXUjJpRdNThG DAjxRfPtDpRTWEpZ1QgzrGt152CQZDntC0twU5s8cCIqgMGmWjbMt0YD0I6kGXc/TEFCQPbRxF+mR 0e/AC1qHE8eC7a13umYJOjfAB4nqadEwYA1mEDBQLRnDkoBcvvp77kdp/BLeJ63UJBOxb1cVynrpD wSOJE1FWSuo5Q3b77qww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1ql9r9-00GdK6-2B; Tue, 26 Sep 2023 15:20:11 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1ql9r7-00GdJk-0n for linux-arm-kernel@lists.infradead.org; Tue, 26 Sep 2023 15:20:10 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id B372960EFB; Tue, 26 Sep 2023 15:20:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 117E9C433C8; Tue, 26 Sep 2023 15:20:05 +0000 (UTC) Date: Tue, 26 Sep 2023 16:20:03 +0100 From: Catalin Marinas To: Shameerali Kolothum Thodi Cc: Oliver Upton , "kvmarm@lists.linux.dev" , "kvm@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "maz@kernel.org" , "will@kernel.org" , "james.morse@arm.com" , "suzuki.poulose@arm.com" , yuzenghui , zhukeqian , Jonathan Cameron , Linuxarm Subject: Re: [RFC PATCH v2 3/8] KVM: arm64: Add some HW_DBM related pgtable interfaces Message-ID: References: <20230825093528.1637-1-shameerali.kolothum.thodi@huawei.com> <20230825093528.1637-4-shameerali.kolothum.thodi@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230926_082009_390602_2CF528D7 X-CRM114-Status: GOOD ( 35.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Sep 25, 2023 at 08:04:39AM +0000, Shameerali Kolothum Thodi wrote: > From: Oliver Upton [mailto:oliver.upton@linux.dev] > > On Fri, Sep 22, 2023 at 04:24:11PM +0100, Catalin Marinas wrote: > > > I was wondering if this interferes with the OS dirty tracking (not the > > > KVM one) but I think that's ok, at least at this point, since the PTE is > > > already writeable and a fault would have marked the underlying page as > > > dirty (user_mem_abort() -> kvm_set_pfn_dirty()). > > > > > > I'm not particularly fond of relying on this but I need to see how it > > > fits with the rest of the series. IIRC KVM doesn't go around and make > > > Stage 2 PTEs read-only but rather unmaps them when it changes the > > > permission of the corresponding Stage 1 VMM mapping. > > > > > > My personal preference would be to track dirty/clean properly as we do > > > for stage 1 (e.g. DBM means writeable PTE) but it has some downsides > > > like the try_to_unmap() code having to retrieve the dirty state via > > > notifiers. > > > > KVM's usage of DBM is complicated by the fact that the dirty log > > interface w/ userspace is at PTE granularity. We only want the page > > table walker to relax PTEs, but take faults on hugepages so we can do > > page splitting. Thanks for the clarification. > > > > @@ -952,6 +990,11 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx, > > > > stage2_pte_executable(new)) > > > > mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); > > > > > > > > + /* Save the possible hardware dirty info */ > > > > + if ((ctx->level == KVM_PGTABLE_MAX_LEVELS - 1) && > > > > + stage2_pte_writeable(ctx->old)) > > > > + mark_page_dirty(kvm_s2_mmu_to_kvm(pgt->mmu), ctx->addr >> PAGE_SHIFT); > > > > + > > > > stage2_make_pte(ctx, new); > > > > > > Isn't this racy and potentially losing the dirty state? Or is the 'new' > > > value guaranteed to have the S2AP[1] bit? For stage 1 we normally make > > > the page genuinely read-only (clearing DBM) in a cmpxchg loop to > > > preserve the dirty state (see ptep_set_wrprotect()). > > > > stage2_try_break_pte() a few lines up does a cmpxchg() and full > > break-before-make, so at this point there shouldn't be a race with > > either software or hardware table walkers. Ah, I missed this. Also it was unrelated to this patch (or rather not introduced by this patch). > > In both cases the 'old' translation should have DBM cleared. Even if the > > PTE were dirty, this is wasted work since we need to do a final scan of > > the stage-2 when userspace collects the dirty log. > > > > Am I missing something? > > I think we can get rid of the above mark_page_dirty(). I will test it to confirm > we are not missing anything here. Is this the case for the other places of mark_page_dirty() in your patches? If stage2_pte_writeable() is true, it must have been made writeable earlier by a fault and the underlying page marked as dirty. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel