From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CA690C48BEB for ; Thu, 15 Feb 2024 11:28:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GyLU796QlokvbUYzOjnw0OxH0LIn5C4O6Cedwhtv69A=; b=fksqZUaLz0fiHp RpshwXlhdRNCunIYmY3hS7SQfIb0KlzUZCbIdUZekwE/FYyvHwbmEK66dOjYoXzKRz4deppMECYKX YCpXu4iKXF4eBCO+cqTNfPZJj0fxN3FcOTjWoLOxhN5XNfstL41Eacg3JOb02EqCODIUCum8Ofk4I yQNA9fiTmU0OIjRpjbbUUIi0Pexx/MO/xPOqpYvyTybvP7EEZDMExF68QsuxW0IZLNfYSq/MDw1DM UnIIx9sOrFuCHv0/yn6Lcmt+SQ7i5Z+qM/5BrrC5sQZGTesX2NpLRL/5gB4ztFnGBfYaCVhdfNqdU Upw8iC3GTKKDDybTRYSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raZuo-0000000G2Qz-3o3g; Thu, 15 Feb 2024 11:28:30 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raZul-0000000G2PF-3wVb for linux-arm-kernel@lists.infradead.org; Thu, 15 Feb 2024 11:28:29 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D2B3ADA7; Thu, 15 Feb 2024 03:29:07 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.68.11]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 522743F7B4; Thu, 15 Feb 2024 03:28:23 -0800 (PST) Date: Thu, 15 Feb 2024 11:28:20 +0000 From: Mark Rutland To: Ryan Roberts Cc: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v6 13/18] arm64/mm: Implement new wrprotect_ptes() batch API Message-ID: References: <20240215103205.2607016-1-ryan.roberts@arm.com> <20240215103205.2607016-14-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240215103205.2607016-14-ryan.roberts@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240215_032828_113916_77A116B1 X-CRM114-Status: GOOD ( 36.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Feb 15, 2024 at 10:32:00AM +0000, Ryan Roberts wrote: > Optimize the contpte implementation to fix some of the fork performance > regression introduced by the initial contpte commit. Subsequent patches > will solve it entirely. > > During fork(), any private memory in the parent must be write-protected. > Previously this was done 1 PTE at a time. But the core-mm supports > batched wrprotect via the new wrprotect_ptes() API. So let's implement > that API and for fully covered contpte mappings, we no longer need to > unfold the contpte. This has 2 benefits: > > - reduced unfolding, reduces the number of tlbis that must be issued. > - The memory remains contpte-mapped ("folded") in the parent, so it > continues to benefit from the more efficient use of the TLB after > the fork. > > The optimization to wrprotect a whole contpte block without unfolding is > possible thanks to the tightening of the Arm ARM in respect to the > definition and behaviour when 'Misprogramming the Contiguous bit'. See > section D21194 at https://developer.arm.com/documentation/102105/ja-07/ > > Tested-by: John Hubbard > Signed-off-by: Ryan Roberts Acked-by: Mark Rutland Mark. > --- > arch/arm64/include/asm/pgtable.h | 61 ++++++++++++++++++++++++++------ > arch/arm64/mm/contpte.c | 38 ++++++++++++++++++++ > 2 files changed, 89 insertions(+), 10 deletions(-) > > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h > index 831099cfc96b..8643227c318b 100644 > --- a/arch/arm64/include/asm/pgtable.h > +++ b/arch/arm64/include/asm/pgtable.h > @@ -978,16 +978,12 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, > } > #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ > > -/* > - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware > - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. > - */ > -static inline void __ptep_set_wrprotect(struct mm_struct *mm, > - unsigned long address, pte_t *ptep) > +static inline void ___ptep_set_wrprotect(struct mm_struct *mm, > + unsigned long address, pte_t *ptep, > + pte_t pte) > { > - pte_t old_pte, pte; > + pte_t old_pte; > > - pte = __ptep_get(ptep); > do { > old_pte = pte; > pte = pte_wrprotect(pte); > @@ -996,6 +992,25 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, > } while (pte_val(pte) != pte_val(old_pte)); > } > > +/* > + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware > + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. > + */ > +static inline void __ptep_set_wrprotect(struct mm_struct *mm, > + unsigned long address, pte_t *ptep) > +{ > + ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); > +} > + > +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, > + pte_t *ptep, unsigned int nr) > +{ > + unsigned int i; > + > + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) > + __ptep_set_wrprotect(mm, address, ptep); > +} > + > #ifdef CONFIG_TRANSPARENT_HUGEPAGE > #define __HAVE_ARCH_PMDP_SET_WRPROTECT > static inline void pmdp_set_wrprotect(struct mm_struct *mm, > @@ -1149,6 +1164,8 @@ extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep); > extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep); > +extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, unsigned int nr); > extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep, > pte_t entry, int dirty); > @@ -1268,12 +1285,35 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, > return contpte_ptep_clear_flush_young(vma, addr, ptep); > } > > +#define wrprotect_ptes wrprotect_ptes > +static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, unsigned int nr) > +{ > + if (likely(nr == 1)) { > + /* > + * Optimization: wrprotect_ptes() can only be called for present > + * ptes so we only need to check contig bit as condition for > + * unfold, and we can remove the contig bit from the pte we read > + * to avoid re-reading. This speeds up fork() which is sensitive > + * for order-0 folios. Equivalent to contpte_try_unfold(). > + */ > + pte_t orig_pte = __ptep_get(ptep); > + > + if (unlikely(pte_cont(orig_pte))) { > + __contpte_try_unfold(mm, addr, ptep, orig_pte); > + orig_pte = pte_mknoncont(orig_pte); > + } > + ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); > + } else { > + contpte_wrprotect_ptes(mm, addr, ptep, nr); > + } > +} > + > #define __HAVE_ARCH_PTEP_SET_WRPROTECT > static inline void ptep_set_wrprotect(struct mm_struct *mm, > unsigned long addr, pte_t *ptep) > { > - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); > - __ptep_set_wrprotect(mm, addr, ptep); > + wrprotect_ptes(mm, addr, ptep, 1); > } > > #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS > @@ -1305,6 +1345,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, > #define ptep_clear_flush_young __ptep_clear_flush_young > #define __HAVE_ARCH_PTEP_SET_WRPROTECT > #define ptep_set_wrprotect __ptep_set_wrprotect > +#define wrprotect_ptes __wrprotect_ptes > #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS > #define ptep_set_access_flags __ptep_set_access_flags > > diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c > index 6d7f40667fa2..bedb58524535 100644 > --- a/arch/arm64/mm/contpte.c > +++ b/arch/arm64/mm/contpte.c > @@ -26,6 +26,26 @@ static inline pte_t *contpte_align_down(pte_t *ptep) > return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); > } > > +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, unsigned int nr) > +{ > + /* > + * Unfold any partially covered contpte block at the beginning and end > + * of the range. > + */ > + > + if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) > + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); > + > + if (ptep + nr != contpte_align_down(ptep + nr)) { > + unsigned long last_addr = addr + PAGE_SIZE * (nr - 1); > + pte_t *last_ptep = ptep + nr - 1; > + > + contpte_try_unfold(mm, last_addr, last_ptep, > + __ptep_get(last_ptep)); > + } > +} > + > static void contpte_convert(struct mm_struct *mm, unsigned long addr, > pte_t *ptep, pte_t pte) > { > @@ -238,6 +258,24 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, > } > EXPORT_SYMBOL(contpte_ptep_clear_flush_young); > > +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, unsigned int nr) > +{ > + /* > + * If wrprotecting an entire contig range, we can avoid unfolding. Just > + * set wrprotect and wait for the later mmu_gather flush to invalidate > + * the tlb. Until the flush, the page may or may not be wrprotected. > + * After the flush, it is guaranteed wrprotected. If it's a partial > + * range though, we must unfold, because we can't have a case where > + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this > + * would cause it to continue to be unpredictable after the flush. > + */ > + > + contpte_try_unfold_partial(mm, addr, ptep, nr); > + __wrprotect_ptes(mm, addr, ptep, nr); > +} > +EXPORT_SYMBOL(contpte_wrprotect_ptes); > + > int contpte_ptep_set_access_flags(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep, > pte_t entry, int dirty) > -- > 2.25.1 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel