From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B40CA137923 for ; Sun, 22 Dec 2024 04:07:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734840476; cv=none; b=KvxenG7qwn0qdiAlkD6AaIU/oKt0j0HTCt2ZbJJwyvrf0FHFiqLh1RCqU/iUlogzYY5qpsq/KfsPkD9R6nFn3npIkjWbPrUCI7nJAJ2ZnT/+dRjGd61qM6tZLdjd8/KON+qghJFWJGGguOUK7GZvzMLHZL15YPuLGyIPLTC/DU4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734840476; c=relaxed/simple; bh=w3coBL+1aqO4OHU8ElST8WMvBbkLlUchTaNILM1cKKY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YaXift4JGFG0fZHOlZRknUVuoYpHs8ePGxy3BYgoYzt5+7Q1ivw6VC/yOxMef1nsxkXRjtpxJEe4qQoIyMElJAX1FwYUh+LjgZdnUfcC/5bImf0Bcj1wODJ8yua2QOP12M5vhID8fGA3CPGR/jm7vlseUyDm+hVmYQk1fZp00ek= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=shelob.surriel.com; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shelob.surriel.com Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tPDFU-000000000V0-0VRF; Sat, 21 Dec 2024 23:07:24 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, akpm@linux-foundation.org, Rik van Riel Subject: [PATCH 09/10] x86/mm: enable AMD translation cache extensions Date: Sat, 21 Dec 2024 23:06:41 -0500 Message-ID: <20241222040717.3096835-10-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241222040717.3096835-1-riel@surriel.com> References: <20241222040717.3096835-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: riel@surriel.com With AMD TCE (translation cache extensions) only the intermediate mappings that cover the address range zapped by INVLPG / INVLPGB get invalidated, rather than all intermediate mappings getting zapped at every TLB invalidation. This can help reduce the TLB miss rate, by keeping more intermediate mappings in the cache. Signed-off-by: Rik van Riel --- arch/x86/kernel/cpu/amd.c | 8 ++++++++ arch/x86/mm/tlb.c | 10 +++++++--- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index 6a6adbe9ae54..34f85aa18fca 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -1143,6 +1143,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) /* Max number of pages INVLPGB can invalidate in one shot */ c->invlpgb_count_max = (edx & 0xffff) + 1; + + /* If supported, enable translation cache extensions (TCE) */ + cpuid(0x80000001, &eax, &ebx, &ecx, &edx); + if (ecx & BIT(17)) { + u64 msr = native_read_msr(MSR_EFER);; + msr |= BIT(15); + wrmsrl(MSR_EFER, msr); + } } static const struct cpu_dev amd_cpu_dev = { diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 0482042e011c..9b13d97d0fb5 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -489,7 +489,7 @@ static void broadcast_tlb_flush(struct flush_tlb_info *info) if (info->stride_shift > PMD_SHIFT) maxnr = 1; - if (info->end == TLB_FLUSH_ALL) { + if (info->end == TLB_FLUSH_ALL || info->freed_tables) { invlpgb_flush_single_pcid(kern_pcid(asid)); /* Do any CPUs supporting INVLPGB need PTI? */ if (static_cpu_has(X86_FEATURE_PTI)) @@ -1122,7 +1122,7 @@ static void flush_tlb_func(void *info) * * The only question is whether to do a full or partial flush. * - * We do a partial flush if requested and two extra conditions + * We do a partial flush if requested and three extra conditions * are met: * * 1. f->new_tlb_gen == local_tlb_gen + 1. We have an invariant that @@ -1149,10 +1149,14 @@ static void flush_tlb_func(void *info) * date. By doing a full flush instead, we can increase * local_tlb_gen all the way to mm_tlb_gen and we can probably * avoid another flush in the very near future. + * + * 3. No page tables were freed. If page tables were freed, a full + * flush ensures intermediate translations in the TLB get flushed. */ if (f->end != TLB_FLUSH_ALL && f->new_tlb_gen == local_tlb_gen + 1 && - f->new_tlb_gen == mm_tlb_gen) { + f->new_tlb_gen == mm_tlb_gen && + !f->freed_tables) { /* Partial flush */ unsigned long addr = f->start; -- 2.47.1