From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ADA29C4345F for ; Sun, 14 Apr 2024 23:25:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WLwQu9+eV86H33Jn837J5dnkbpZv669po+f4COkXUks=; b=2iNhvfXWoy5oh1 9s6szJu/SrQgqRfhlo2LW2xG9p99ncpmQpg5OIASPbGJt68BeyBavwM70LWHNL63+rc+3lMewxvFP wpbbNzFua7ECkE4bB+hc38ttP9TW9cI9XH1q44jRj7nbgac+CgUL6vDi/XYa/YHfNG1qpAyCe3sQJ PtzMPtlYfUvi7YYloI1XIMu1DU0my/ch0Wj61feRB6NVQWBW+CIVZELkB/ZiTzHjrDRxSILyJAin1 fCZJD+ZoB23dl8vVLXzkASheMAvHhrLyx9myAJZCbQT3DSmJZxjMONHmmOd2DEepRl48drqSwsWVg di2WrJjt+/UZro9/d+/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rw9Dm-00000006Xop-0KlM; Sun, 14 Apr 2024 23:25:14 +0000 Received: from out-172.mta1.migadu.com ([95.215.58.172]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rw9Df-00000006XlG-3myO for linux-arm-kernel@lists.infradead.org; Sun, 14 Apr 2024 23:25:12 +0000 Date: Thu, 11 Apr 2024 23:03:47 +0900 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1713137091; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kPKj/T41SmjIhbsDiYtc1158xq+evUdUQS9S8ehmiTE=; b=oEUkGyoPo7WPHZVoow2FKmC2OkbJ459rmVATb2AvW0obDmr96lbEFRzG6Z3zAZG7ifdOks lhSXdkk16ohdWiDFHAP5P2RIv1vwwygkcippWLznji4bpnk4PXC7qYGsJnfh55S1/6kjfE 2e2iCNQv4kYS8foJeJmEMWBE+HOdVr8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Itaru Kitayama To: Ryan Roberts Cc: Catalin Marinas , Will Deacon , Mark Rutland , Ard Biesheuvel , David Hildenbrand , Donald Dutile , Eric Chanudet , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Itaru Kitayama Subject: Re: [PATCH v3 1/3] arm64: mm: Don't remap pgtables per-cont(pte|pmd) block Message-ID: References: <20240412131908.433043-1-ryan.roberts@arm.com> <20240412131908.433043-2-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240412131908.433043-2-ryan.roberts@arm.com> X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240414_162508_483880_124BF6F2 X-CRM114-Status: GOOD ( 25.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Ryan, On Fri, Apr 12, 2024 at 02:19:06PM +0100, Ryan Roberts wrote: > A large part of the kernel boot time is creating the kernel linear map > page tables. When rodata=full, all memory is mapped by pte. And when > there is lots of physical ram, there are lots of pte tables to populate. > The primary cost associated with this is mapping and unmapping the pte > table memory in the fixmap; at unmap time, the TLB entry must be > invalidated and this is expensive. > > Previously, each pmd and pte table was fixmapped/fixunmapped for each > cont(pte|pmd) block of mappings (16 entries with 4K granule). This means > we ended up issuing 32 TLBIs per (pmd|pte) table during the population > phase. > > Let's fix that, and fixmap/fixunmap each page once per population, for a > saving of 31 TLBIs per (pmd|pte) table. This gives a significant boot > speedup. > > Execution time of map_mem(), which creates the kernel linear map page > tables, was measured on different machines with different RAM configs: > > | Apple M2 VM | Ampere Altra| Ampere Altra| Ampere Altra > | VM, 16G | VM, 64G | VM, 256G | Metal, 512G > ---------------|-------------|-------------|-------------|------------- > | ms (%) | ms (%) | ms (%) | ms (%) > ---------------|-------------|-------------|-------------|------------- > before | 168 (0%) | 2198 (0%) | 8644 (0%) | 17447 (0%) > after | 78 (-53%) | 435 (-80%) | 1723 (-80%) | 3779 (-78%) > > Signed-off-by: Ryan Roberts > Tested-by: Itaru Kitayama > Tested-by: Eric Chanudet > --- > arch/arm64/mm/mmu.c | 27 ++++++++++++++------------- > 1 file changed, 14 insertions(+), 13 deletions(-) > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c > index 495b732d5af3..9f1d69b7b494 100644 > --- a/arch/arm64/mm/mmu.c > +++ b/arch/arm64/mm/mmu.c > @@ -172,12 +172,9 @@ bool pgattr_change_is_safe(u64 old, u64 new) > return ((old ^ new) & ~mask) == 0; > } > > -static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, > +static void init_pte(pte_t *ptep, unsigned long addr, unsigned long end, > phys_addr_t phys, pgprot_t prot) > { > - pte_t *ptep; > - > - ptep = pte_set_fixmap_offset(pmdp, addr); > do { > pte_t old_pte = __ptep_get(ptep); > > @@ -192,8 +189,6 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, > > phys += PAGE_SIZE; > } while (ptep++, addr += PAGE_SIZE, addr != end); > - > - pte_clear_fixmap(); > } > > static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, > @@ -204,6 +199,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, > { > unsigned long next; > pmd_t pmd = READ_ONCE(*pmdp); > + pte_t *ptep; > > BUG_ON(pmd_sect(pmd)); > if (pmd_none(pmd)) { > @@ -219,6 +215,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, > } > BUG_ON(pmd_bad(pmd)); > > + ptep = pte_set_fixmap_offset(pmdp, addr); > do { > pgprot_t __prot = prot; > > @@ -229,20 +226,21 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, > (flags & NO_CONT_MAPPINGS) == 0) > __prot = __pgprot(pgprot_val(prot) | PTE_CONT); > > - init_pte(pmdp, addr, next, phys, __prot); > + init_pte(ptep, addr, next, phys, __prot); > > + ptep += pte_index(next) - pte_index(addr); > phys += next - addr; > } while (addr = next, addr != end); > + > + pte_clear_fixmap(); > } > > -static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, > +static void init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, > phys_addr_t phys, pgprot_t prot, > phys_addr_t (*pgtable_alloc)(int), int flags) > { > unsigned long next; > - pmd_t *pmdp; > > - pmdp = pmd_set_fixmap_offset(pudp, addr); > do { > pmd_t old_pmd = READ_ONCE(*pmdp); > > @@ -268,8 +266,6 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, > } > phys += next - addr; > } while (pmdp++, addr = next, addr != end); > - > - pmd_clear_fixmap(); > } > > static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, > @@ -279,6 +275,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, > { > unsigned long next; > pud_t pud = READ_ONCE(*pudp); > + pmd_t *pmdp; > > /* > * Check for initial section mappings in the pgd/pud. > @@ -297,6 +294,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, > } > BUG_ON(pud_bad(pud)); > > + pmdp = pmd_set_fixmap_offset(pudp, addr); > do { > pgprot_t __prot = prot; > > @@ -307,10 +305,13 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, > (flags & NO_CONT_MAPPINGS) == 0) > __prot = __pgprot(pgprot_val(prot) | PTE_CONT); > > - init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); > + init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags); > > + pmdp += pmd_index(next) - pmd_index(addr); > phys += next - addr; > } while (addr = next, addr != end); > + > + pmd_clear_fixmap(); > } > > static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, I looked at this specific patch 1/3 for a while and now make sense the code changes to reduce down the number of TLBIs respecting the contigous bit where available at both PMD and PTE levels. I can not finish other 2/3 and 3/3 patches in a timely manner but I'd like to give my Reviewied-by: Itaru Kitayama on 1/3. Thanks, Itaru. > -- > 2.25.1 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel