From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6864C7EE2C for ; Wed, 24 May 2023 10:18:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+fCbYEIICVJ8PV+iFXbYG/Ms40dXTcQv5gdxmcbysPw=; b=S0k8Ur1q1C0Fp3 NNOWkWUF9jX+m5x/ccZflF++OmPbRnOpQO3S6egYQ4nFVpOuAYwZJ85Dy7N1OcRUxXxC3bK66MHdt 6AJbMnzG/kZKluzGzhqzVlKR9R3WSLGAmmvbdiXmhkMDLZ7VN+NivAvhI98PM1Ditie75qk7t8Z3z tTkKkRV11Drxv+XjfXSClyTXOiHRcEHKBByCudilIsVpCqgPt9FP7Is2I3dnJHLZjRABrJQsnHcb0 ZvDDRfq4SHPxGhIIkCoNqKFpcr26Gfjq8PoywBScRuAr/JUf+yv6SuREp92SFvnpWKdliFoybZMhr sdWBe7ocnTi7Aj014W/g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q1lZX-00D4t9-31; Wed, 24 May 2023 10:18:23 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q1lZV-00D4sC-05 for linux-arm-kernel@lists.infradead.org; Wed, 24 May 2023 10:18:22 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BB46C1042; Wed, 24 May 2023 03:19:02 -0700 (PDT) Received: from [10.57.84.6] (unknown [10.57.84.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CE0793F67D; Wed, 24 May 2023 03:18:16 -0700 (PDT) Message-ID: <14ca0866-cdf1-736c-409b-7318bdfba71f@arm.com> Date: Wed, 24 May 2023 11:18:12 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: [PATCH] ARM: tlb: Prevent flushing insane large ranges one by one To: Thomas Gleixner , linux-arm-kernel@lists.infradead.org Cc: Russell King , John Ogness , Arnd Bergmann References: <87pm6qm5wo.ffs@tglx> Content-Language: en-GB From: Robin Murphy In-Reply-To: <87pm6qm5wo.ffs@tglx> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230524_031821_181678_059661CF X-CRM114-Status: GOOD ( 31.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2023-05-24 10:32, Thomas Gleixner wrote: > vmalloc uses lazy TLB flushes for unmapped ranges to avoid excessive TLB > flushing on every unmap. The lazy flushing coalesces unmapped ranges and > invokes flush_tlb_kernel_range() with the combined range. > > The coalescing can result in ranges which spawn the full vmalloc address > range. In the case of flushing an executable mapping in the module address > space this range is extended to also flush the direct map alias. > > flush_tlb_kernel_range() then walks insane large ranges, the worst case > observed was ~1.5GB. > > The range is flushed page by page, which takes several milliseconds to > complete in the worst case and obviously affects all processes in the > system. In the worst case observed this causes the runtime of a realtime > task on an isolated CPU to be almost doubled over the normal worst > case, which makes it miss the deadline. > > Cure this by sanity checking the range against a threshold and fall back to > tlb_flush_all() when the range is too large. > > The default threshold is 32 pages, but for CPUs with CP15 this is evaluated > at boot time via read_cpuid(CPUID_TLBTYPE) and set to the half of the TLB > size. > > The vmalloc range coalescing could be improved to provide a list or > array of ranges to flush, which allows to avoid overbroad flushing, but > that's a major surgery and does not solve the problem of actual > justified large range flushes which can happen due to the lazy flush > mechanics in vmalloc. The lazy flush results in batching which is biased > towards large range flushes by design. > > Fixes: db64fe02258f ("mm: rewrite vmap layer") > Reported-by: John Ogness > Debugged-by: John Ogness > Signed-off-by: Thomas Gleixner > Tested-by: John Ogness > Link: https://lore.kernel.org/all/87a5y5a6kj.ffs@tglx > --- > arch/arm/include/asm/cputype.h | 5 +++++ > arch/arm/include/asm/tlbflush.h | 2 ++ > arch/arm/kernel/setup.c | 10 ++++++++++ > arch/arm/kernel/smp_tlb.c | 4 ++++ > 4 files changed, 21 insertions(+) > > --- a/arch/arm/include/asm/cputype.h > +++ b/arch/arm/include/asm/cputype.h > @@ -196,6 +196,11 @@ static inline unsigned int __attribute_c > return read_cpuid(CPUID_MPUIR); > } > > +static inline unsigned int __attribute_const__ read_cpuid_tlbsize(void) > +{ > + return 64 << ((read_cpuid(CPUID_TLBTYPE) >> 1) & 0x03); > +} This appears to be specific to Cortex-A9 - these bits are implementation-defined, and it looks like on on most other Arm Ltd. CPUs they have no meaning at all, e.g.[1][2][3], but they could still hold some wildly unrelated value on other implementations. Thanks, Robin. [1] https://developer.arm.com/documentation/ddi0344/k/system-control-coprocessor/system-control-coprocessor-registers/c0--tlb-type-register [2] https://developer.arm.com/documentation/ddi0464/f/System-Control/Register-descriptions/TLB-Type-Register [3] https://developer.arm.com/documentation/ddi0500/j/System-Control/AArch32-register-descriptions/TLB-Type-Register > + > #elif defined(CONFIG_CPU_V7M) > > static inline unsigned int __attribute_const__ read_cpuid_id(void) > --- a/arch/arm/include/asm/tlbflush.h > +++ b/arch/arm/include/asm/tlbflush.h > @@ -210,6 +210,8 @@ struct cpu_tlb_fns { > unsigned long tlb_flags; > }; > > +extern unsigned int tlb_flush_all_threshold; > + > /* > * Select the calling method > */ > --- a/arch/arm/kernel/setup.c > +++ b/arch/arm/kernel/setup.c > @@ -90,6 +90,8 @@ EXPORT_SYMBOL(__machine_arch_type); > unsigned int cacheid __read_mostly; > EXPORT_SYMBOL(cacheid); > > +unsigned int tlb_flush_all_threshold __ro_after_init = 32; > + > unsigned int __atags_pointer __initdata; > > unsigned int system_rev; > @@ -356,6 +358,13 @@ static void __init cacheid_init(void) > cache_is_vipt_nonaliasing() ? "VIPT nonaliasing" : "unknown"); > } > > +static void __init tlbsize_init(void) > +{ > +#ifdef CONFIG_CPU_CP15 > + tlb_flush_all_threshold = read_cpuid_tlbsize() / 2; > +#endif > +} > + > /* > * These functions re-use the assembly code in head.S, which > * already provide the required functionality. > @@ -747,6 +756,7 @@ static void __init setup_processor(void) > elf_hwcap_fixup(); > > cacheid_init(); > + tlbsize_init(); > cpu_init(); > } > > --- a/arch/arm/kernel/smp_tlb.c > +++ b/arch/arm/kernel/smp_tlb.c > @@ -234,6 +234,10 @@ void flush_tlb_range(struct vm_area_stru > > void flush_tlb_kernel_range(unsigned long start, unsigned long end) > { > + if ((end - start) > (tlb_flush_all_threshold << PAGE_SHIFT)) { > + flush_tlb_all(); > + return; > + } > if (tlb_ops_need_broadcast()) { > struct tlb_args ta; > ta.ta_start = start; > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel