From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFB831F3B8A for ; Sat, 30 Aug 2025 03:54:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756526048; cv=none; b=iv05AkhhNAE2gRxO7VJ3BmRwM6diERElzorzlYvMgKw6ttc4zLyFXekilOsKDPLt8z4L1ksv4jD/2509/ajs1CgoNX0ImNV4YorElt8Wp2aeXmGWB7tHM05iywdol2NF9BpRsG8gl+gEjSHhz4Ud2tbt7kSNHShUEPigFQAlIHk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756526048; c=relaxed/simple; bh=2xc8YLH0JQH4SyWzhUqZgF59rBHDu1OBV/U5hrqhdbg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=sM6ZxjGse1wzKuTF9xT3B4XZAHDdm0j8TjkIVEgTpZGQECyeeLZ0fca5KxbKTwTHKggNmnU5uqBND3ZIoxnBXSwmcAXfIDhbcs/lF/QNDswTZctTXOTnYdGf6GcXCoehq2b/cpDrKZucREANY063Pu4oHor4m90Yw7Gg6H7mUVQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=T5sdDp66; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="T5sdDp66" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-32326e09f58so2793262a91.2 for ; Fri, 29 Aug 2025 20:54:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756526046; x=1757130846; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=91RPs/0fYM/0BKV/CtBkeG0ygHgvI5Jam/unEho/Uss=; b=T5sdDp66ky7TZA8Ql/Tox1UccAHW/pZhdMMjqDjN98QtoHKEueP2KIqrPQBaGu3N2U 0Z35QHpDBXVk+xQZKyoiuN5igvfqSbIVjp4mR5fQCWRWqe6gbqDQu+60rQn7QVx4KXmj JAqdx5YZU82a6MEewxO7cpTvArvTCfzfXDK6fQSqH0dgeQq/0EkL85Z7Ki74sW2h6FFk AYDUkz3l+A5DsOqYid6vRsRGzRlKxQwiWGAK1XVCgcUQiezVJQN1TqRXF6mabehmyyyj cvyXH2udMeT6rIVbqEKf1MER2IDzi8dcMU7VTRpFYMoT+M8My+CiqCIDlP6dIPxHvVIq v3gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756526046; x=1757130846; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=91RPs/0fYM/0BKV/CtBkeG0ygHgvI5Jam/unEho/Uss=; b=o6+jJlE/EZn0W9AUWdr2CeWxQEgtpE8UU33uwmbGfpjVg1XzC1KNNroG2Fk0Rb9LZd Lnstn8p2egL/mVR2JKU+mgztCMHq1NTMsKZS9PmYFEcB9idTB71gpUMk69IAjd77mlC1 sXL41FWYhNXZBioJrAJHLbpBUBNST3F+ZgFq/RsL6AQ8vrCPncsqwZzpteJs2bQ2JBTz 1oA+qxwbw0OiMmwox1T9eK4si6ZJjS4k3ngO2aLYegFRei+yO6B8PpbB+Sjuq6QovfaR n+SGTduX1qweboq9s9zddDFD9eqC7DzeFvT/oLM6zVdxVMc/EswwZH3i08bAucsTAIgS WgfQ== X-Forwarded-Encrypted: i=1; AJvYcCW6f0uUmkm+ENXSdsceB4raa6bKH09JJWUYg8slxxugKjV7ukKptTbgiQulobfwXltMpiK0rZk=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6xeCHx43EoXv+sbf/BwwOaO/YZ0o1AYEf63MqzCW9aUsO4T0c YeCK+QyJmLu3H25PowTCL/T6qiVLbOJFLuZF709v35QGBDgDm18yWi8A X-Gm-Gg: ASbGnctRdkNt3JttLBgIm1zruT6VUQHWiEmPtJCoYoXPVIiJ/GLxsCyZ1eOXWgx7fbl 1vhNMCdAq3zFdts9algMapFqBQnoxV7gWsnJ5Ms4G6FuhaYoBv/lnYwVtxgA2wFYgVah3m7Gm4U QovYpP+tLuSkG4Q8GFbEhrM2yPJBBpVRDq8YavvEIS0CKsDEc6oAgFWoLtdIkMgzIBtfY5KK9qS H+PPh2a9csHZHJZEEsNUKL2cHS3nlvTuKnWV+hE4rgE+dRjOOfoXsrJfAyEA+sE3IP938/+XZ/l GO++xcGQlv0TuPWSp3+RAamArSqnTLYI3V4YiGlwR9eL4Gf/3k0VznvShPg+NAEtMEdtDk9LhEx lAoYRSM1mBs3AJRpltn8qDm+XWw== X-Google-Smtp-Source: AGHT+IErkqyMgcM5O4XsY4PE7xy0Jnw7MTzY3U7M20aCxbjJOga65xvpy+dBG2J3XXIo8tmY5uvLLA== X-Received: by 2002:a17:90b:2751:b0:312:e731:5a66 with SMTP id 98e67ed59e1d1-32815412c9emr1406893a91.3.1756526045848; Fri, 29 Aug 2025 20:54:05 -0700 (PDT) Received: from dw-tp ([171.76.86.139]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-327da90ee17sm4279879a91.24.2025.08.29.20.54.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Aug 2025 20:54:05 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev@lists.ozlabs.org Cc: Donet Tom , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Mackerras , "Aneesh Kumar K.V" , stable@vger.kernel.org Subject: [RFC 1/8] powerpc/mm: Fix SLB multihit issue during SLB preload Date: Sat, 30 Aug 2025 09:21:40 +0530 Message-ID: <6378feb7f4aeb8353905926a63c938022cf07f6c.1756522067.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: stable@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From: Donet Tom On systems using the hash MMU, there is a software SLB preload cache that mirrors the entries loaded into the hardware SLB buffer. This preload cache is subject to periodic eviction — typically after every 256 context switches — to remove old entry. To optimize performance, the kernel skips switch_mmu_context() in switch_mm_irqs_off() when the prev and next mm_struct are the same. However, on hash MMU systems, this can lead to inconsistencies between the hardware SLB and the software preload cache. If an SLB entry for a process is evicted from the software cache on one CPU, and the same process later runs on another CPU without executing switch_mmu_context(), the hardware SLB may retain stale entries. If the kernel then attempts to reload that entry, it can trigger an SLB multi-hit error. The following timeline shows how stale SLB entries are created and can cause a multi-hit error when a process moves between CPUs without a MMU context switch. CPU 0 CPU 1 ----- ----- Process P exec swapper/1 load_elf_binary begin_new_exc activate_mm switch_mm_irqs_off switch_mmu_context switch_slb /* * This invalidates all * the entries in the HW * and setup the new HW * SLB entries as per the * preload cache. */ context_switch sched_migrate_task migrates process P to cpu-1 Process swapper/0 context switch (to process P) (uses mm_struct of Process P) switch_mm_irqs_off() switch_slb load_slb++ /* * load_slb becomes 0 here * and we evict an entry from * the preload cache with * preload_age(). We still * keep HW SLB and preload * cache in sync, that is * because all HW SLB entries * anyways gets evicted in * switch_slb during SLBIA. * We then only add those * entries back in HW SLB, * which are currently * present in preload_cache * (after eviction). */ load_elf_binary continues... setup_new_exec() slb_setup_new_exec() sched_switch event sched_migrate_task migrates process P to cpu-0 context_switch from swapper/0 to Process P switch_mm_irqs_off() /* * Since both prev and next mm struct are same we don't call * switch_mmu_context(). This will cause the HW SLB and SW preload * cache to go out of sync in preload_new_slb_context. Because there * was an SLB entry which was evicted from both HW and preload cache * on cpu-1. Now later in preload_new_slb_context(), when we will try * to add the same preload entry again, we will add this to the SW * preload cache and then will add it to the HW SLB. Since on cpu-0 * this entry was never invalidated, hence adding this entry to the HW * SLB will cause a SLB multi-hit error. */ load_elf_binary continues... START_THREAD start_thread preload_new_slb_context /* * This tries to add a new EA to preload cache which was earlier * evicted from both cpu-1 HW SLB and preload cache. This caused the * HW SLB of cpu-0 to go out of sync with the SW preload cache. The * reason for this was, that when we context switched back on CPU-0, * we should have ideally called switch_mmu_context() which will * bring the HW SLB entries on CPU-0 in sync with SW preload cache * entries by setting up the mmu context properly. But we didn't do * that since the prev mm_struct running on cpu-0 was same as the * next mm_struct (which is true for swapper / kernel threads). So * now when we try to add this new entry into the HW SLB of cpu-0, * we hit a SLB multi-hit error. */ WARNING: CPU: 0 PID: 1810970 at arch/powerpc/mm/book3s64/slb.c:62 assert_slb_presence+0x2c/0x50(48 results) 02:47:29 [20157/42149] Modules linked in: CPU: 0 UID: 0 PID: 1810970 Comm: dd Not tainted 6.16.0-rc3-dirty #12 VOLUNTARY Hardware name: IBM pSeries (emulated by qemu) POWER8 (architected) 0x4d0200 0xf000004 of:SLOF,HEAD hv:linux,kvm pSeries NIP: c00000000015426c LR: c0000000001543b4 CTR: 0000000000000000 REGS: c0000000497c77e0 TRAP: 0700 Not tainted (6.16.0-rc3-dirty) MSR: 8000000002823033 CR: 28888482 XER: 00000000 CFAR: c0000000001543b0 IRQMASK: 3 <...> NIP [c00000000015426c] assert_slb_presence+0x2c/0x50 LR [c0000000001543b4] slb_insert_entry+0x124/0x390 Call Trace: 0x7fffceb5ffff (unreliable) preload_new_slb_context+0x100/0x1a0 start_thread+0x26c/0x420 load_elf_binary+0x1b04/0x1c40 bprm_execve+0x358/0x680 do_execveat_common+0x1f8/0x240 sys_execve+0x58/0x70 system_call_exception+0x114/0x300 system_call_common+0x160/0x2c4 >>From the above analysis, during early exec the hardware SLB is cleared, and entries from the software preload cache are reloaded into hardware by switch_slb. However, preload_new_slb_context and slb_setup_new_exec also attempt to load some of the same entries, which can trigger a multi-hit. In most cases, these additional preloads simply hit existing entries and add nothing new. Removing these functions avoids redundant preloads and eliminates the multi-hit issue. This patch removes these two functions. We tested process switching performance using the context_switch benchmark on POWER9/hash, and observed no regression. Without this patch: 129041 ops/sec With this patch: 129341 ops/sec We also measured SLB faults during boot, and the counts are essentially the same with and without this patch. SLB faults without this patch: 19727 SLB faults with this patch: 19786 Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Paul Mackerras Cc: "Aneesh Kumar K.V" Cc: Donet Tom Cc: linuxppc-dev@lists.ozlabs.org Fixes: 5434ae74629a ("powerpc/64s/hash: Add a SLB preload cache") cc: Suggested-by: Nicholas Piggin Signed-off-by: Donet Tom --- arch/powerpc/include/asm/book3s/64/mmu-hash.h | 1 - arch/powerpc/kernel/process.c | 5 -- arch/powerpc/mm/book3s64/internal.h | 2 - arch/powerpc/mm/book3s64/mmu_context.c | 2 - arch/powerpc/mm/book3s64/slb.c | 88 ------------------- 5 files changed, 98 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h index 1c4eebbc69c9..e1f77e2eead4 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h +++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h @@ -524,7 +524,6 @@ void slb_save_contents(struct slb_entry *slb_ptr); void slb_dump_contents(struct slb_entry *slb_ptr); extern void slb_vmalloc_update(void); -void preload_new_slb_context(unsigned long start, unsigned long sp); #ifdef CONFIG_PPC_64S_HASH_MMU void slb_set_size(u16 size); diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c index 855e09886503..2b9799157eb4 100644 --- a/arch/powerpc/kernel/process.c +++ b/arch/powerpc/kernel/process.c @@ -1897,8 +1897,6 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) return 0; } -void preload_new_slb_context(unsigned long start, unsigned long sp); - /* * Set up a thread for executing a new program */ @@ -1906,9 +1904,6 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp) { #ifdef CONFIG_PPC64 unsigned long load_addr = regs->gpr[2]; /* saved by ELF_PLAT_INIT */ - - if (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && !radix_enabled()) - preload_new_slb_context(start, sp); #endif #ifdef CONFIG_PPC_TRANSACTIONAL_MEM diff --git a/arch/powerpc/mm/book3s64/internal.h b/arch/powerpc/mm/book3s64/internal.h index a57a25f06a21..c26a6f0c90fc 100644 --- a/arch/powerpc/mm/book3s64/internal.h +++ b/arch/powerpc/mm/book3s64/internal.h @@ -24,8 +24,6 @@ static inline bool stress_hpt(void) void hpt_do_stress(unsigned long ea, unsigned long hpte_group); -void slb_setup_new_exec(void); - void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush); #endif /* ARCH_POWERPC_MM_BOOK3S64_INTERNAL_H */ diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c index 4e1e45420bd4..fb9dcf9ca599 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -150,8 +150,6 @@ static int hash__init_new_context(struct mm_struct *mm) void hash__setup_new_exec(void) { slice_setup_new_exec(); - - slb_setup_new_exec(); } #else static inline int hash__init_new_context(struct mm_struct *mm) diff --git a/arch/powerpc/mm/book3s64/slb.c b/arch/powerpc/mm/book3s64/slb.c index 6b783552403c..7e053c561a09 100644 --- a/arch/powerpc/mm/book3s64/slb.c +++ b/arch/powerpc/mm/book3s64/slb.c @@ -328,94 +328,6 @@ static void preload_age(struct thread_info *ti) ti->slb_preload_tail = (ti->slb_preload_tail + 1) % SLB_PRELOAD_NR; } -void slb_setup_new_exec(void) -{ - struct thread_info *ti = current_thread_info(); - struct mm_struct *mm = current->mm; - unsigned long exec = 0x10000000; - - WARN_ON(irqs_disabled()); - - /* - * preload cache can only be used to determine whether a SLB - * entry exists if it does not start to overflow. - */ - if (ti->slb_preload_nr + 2 > SLB_PRELOAD_NR) - return; - - hard_irq_disable(); - - /* - * We have no good place to clear the slb preload cache on exec, - * flush_thread is about the earliest arch hook but that happens - * after we switch to the mm and have already preloaded the SLBEs. - * - * For the most part that's probably okay to use entries from the - * previous exec, they will age out if unused. It may turn out to - * be an advantage to clear the cache before switching to it, - * however. - */ - - /* - * preload some userspace segments into the SLB. - * Almost all 32 and 64bit PowerPC executables are linked at - * 0x10000000 so it makes sense to preload this segment. - */ - if (!is_kernel_addr(exec)) { - if (preload_add(ti, exec)) - slb_allocate_user(mm, exec); - } - - /* Libraries and mmaps. */ - if (!is_kernel_addr(mm->mmap_base)) { - if (preload_add(ti, mm->mmap_base)) - slb_allocate_user(mm, mm->mmap_base); - } - - /* see switch_slb */ - asm volatile("isync" : : : "memory"); - - local_irq_enable(); -} - -void preload_new_slb_context(unsigned long start, unsigned long sp) -{ - struct thread_info *ti = current_thread_info(); - struct mm_struct *mm = current->mm; - unsigned long heap = mm->start_brk; - - WARN_ON(irqs_disabled()); - - /* see above */ - if (ti->slb_preload_nr + 3 > SLB_PRELOAD_NR) - return; - - hard_irq_disable(); - - /* Userspace entry address. */ - if (!is_kernel_addr(start)) { - if (preload_add(ti, start)) - slb_allocate_user(mm, start); - } - - /* Top of stack, grows down. */ - if (!is_kernel_addr(sp)) { - if (preload_add(ti, sp)) - slb_allocate_user(mm, sp); - } - - /* Bottom of heap, grows up. */ - if (heap && !is_kernel_addr(heap)) { - if (preload_add(ti, heap)) - slb_allocate_user(mm, heap); - } - - /* see switch_slb */ - asm volatile("isync" : : : "memory"); - - local_irq_enable(); -} - static void slb_cache_slbie_kernel(unsigned int index) { unsigned long slbie_data = get_paca()->slb_cache[index]; -- 2.50.1