From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794064997; b=dT/yoWL0zLSJHhFwegTkXGWUYxwG1dkA29+JmfBSRdz+WZ7vMpiQJxeOPhD/xewDGyt/ CtKCv4q3KPMMRSNv/BWb7FEZmzFkBwnfqynkZN4/UaDzub/i69CMax5QjxienX9u9/F+G Sesa+JCkSOr/AufkDi+BxZZYZUPrtEuNI8g5RNx4F5ymUXV5axBr2ObuJlvYsiCyZGqog rxrMTEXh6bsN80aVEtMYaHTNFCMy6Niq2FwG8no7s9DY4pKmy2D2NrdTC3xzjVtfmqcj1 IKDOFW0/hl0DWKux+4ig9j2uRftFaq/6eiISk/EM9mvSPXaDfmQE9HoJ1WCasqJcvgg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794064997; h=Received:DKIM-Signature:From:To:Cc:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:User-Agent:X-stable:X-Patchwork-Hint: MIME-Version:Content-Transfer-Encoding; bh=AVFrlhXv9NbpMj8jIX1PI+jqZ9Q56DcWYBe7JowTHNY=; b=2v3uyy+zqyumkoQlwN4jRDZfNF6c83AIeXE0b1Pa5q1yOb5V4fCtpVpvvbhdBxT2Pv9W 43YbbpJ1KMp1O05E2uY1QHL/UNnNipgg0VAqYo1un7rpzou4q5foLF9ROx28WSYmtZhTa RHWhrqcydj9KbIKStNGnTb+0LwDbcw+yNRo49NlvV1kizQsp45XiSlr7v+R1EPFJ4RCX9 T4bhXyPYWL8v4LroH9guVJsN79p5Zke5tslIHzsBSpv/bgQEMFoh4kRRkNTKx2Es3VGeC P1Wlhj/FFX0v7/FfAapGl0JP0Xe66zSwQ0FTXbMGbSd/9AfS5l3mg8nnWwJ4RQ4mGfQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=SSNVIKx5; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 809E9606A9C3; Sat, 7 Nov 2026 15:23:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=SSNVIKx5 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 280E21F0089F; Sat, 7 Nov 2026 15:23:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1794064997; bh=AVFrlhXv9NbpMj8jIX1PI+jqZ9Q56DcWYBe7JowTHNY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=SSNVIKx5iy5PAYF73fNU5r4q99sQUMFVQyZmNcnpxnaZDzejtoqbfYSY7L1YL/tGk j5KZQDIuXHJ/H4D4fO2seITApiLbqSPjmiqtyrRnWJUiISrRdIGNb8ljajsH7Pem18 sV6JRKClBke4dD4TXwGA2c6wrg2Qt3CIFJgIZAeM= From: Greg Kroah-Hartman To: stable@vger.kernel.org Cc: Greg Kroah-Hartman , patches@lists.linux.dev, Matt Turner , Magnus Lindholm , Sasha Levin Subject: [PATCH 7.3 0867/1257] alpha: do not skip TLB shootdown IPIs for kthread-borrowed mms Date: Sat, 7 Nov 2026 08:52:43 -0500 Message-ID: <20261107135003.271282936@linuxfoundation.org> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261107134939.434104957@linuxfoundation.org> References: <20261107134939.434104957@linuxfoundation.org> User-Agent: quilt/0.69 X-stable: review X-Patchwork-Hint: ignore X-Mailing-List: stable@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit 7.3-stable review patch. If anyone has any objections, please let me know. ------------------ From: Magnus Lindholm [ Upstream commit fe221b7b2d3ecd042dd391f74ff6a6fdeb7c2038 ] flush_tlb_mm(), flush_tlb_page() and flush_icache_user_page() skip the shootdown IPI when mm_users <= 1, on the assumption that no other CPU can be running the mm. A task that borrows an mm through kthread_use_mm() takes mmgrab() rather than mmget(), so it never appears in mm_users, and kthread_use_mm() may be handed an mm that is already the caller's active_mm and loaded on that CPU. Such a CPU was not only left without an IPI, its mm->context[cpu] was cleared from under it. mm->context[cpu] is already the record of which CPUs hold an ASN for the mm. Test it rather than clearing it: take the shortcut only when no other CPU has one, and otherwise fall through to the IPI, which invalidates those CPUs properly. A CPU that merely ran the mm in the past also holds a context and now costs an IPI, which errs in the safe direction. Change ipi_flush_tlb_mm(), ipi_flush_icache_page() and the migration callback ipi_flush_mm_and_page() to test current->mm, like ipi_flush_tlb_page(). A CPU that only retains the mm lazily then clears its own slot through flush_tlb_other(), rather than publishing a new ASN on every IPI or migration rendezvous. The migration callback still does its per-VA tbi() afterwards. Returning to the mm allocates a fresh ASN through ev5_switch_mm(), or through the direct-load path when kthread_use_mm() switches current's mm. Once these historical slots are gone, the shortcut is available again if mm_users remains at most one and no other CPU has taken or kept a context. An explicit kthread_use_mm() borrower has current->mm set and still receives the invalidate it needs. The caller-side tests in flush_tlb_mm() and flush_icache_user_page() remain current->active_mm. A kernel thread that flushes its lazily held mm can therefore publish its own context slot. Those branches also perform the shortcut check, which excludes the calling CPU's slot, so leave them unchanged. This is distinct from a remote lazy CPU keeping its slot populated through repeated callbacks. Together with the preceding patch, dropping these clearing loops leaves every runtime write to mm->context[] targeting the writing CPU's own slot, so the array becomes a lockless publication of which CPUs may be using the mm, with a single writer per slot. Mark the two stores that are now observed from other CPUs; flush_tlb_other() already uses WRITE_ONCE(). The uniprocessor flush_icache_user_page() is left alone, as nothing reads another CPU's slot there, and init_new_context() runs before the mm is shared. Order the slot reads after the page-table changes with smp_mb(). On the publishing CPU, the store in ev5_switch_mm() precedes the mb() in Alpha's arch_spin_unlock(), when finish_lock_switch() drops the rq lock. This relies on Alpha's unlock implementation, not a generic scheduler promise of a full barrier on unlock; document that reliance at the store. Also issue smp_mb() after publishing in __load_new_mm_context(), before loading and using the mm. This covers every caller, including check_mmu_context(), which can publish a replacement after finish_lock_switch() has already run. This requires the preceding removal of remote slot clearing from migrate_flush_tlb_page(). It also requires the stale-TLB series to set need_new_asn on both the reused and newly allocated ASN paths. Otherwise an IPI between finish_lock_switch() and check_mmu_context() can zero a new slot while asn_lock is set, and the task can return to user space with a live ASN that the shortcut cannot see. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Cc: # 53ab52cfe5ce: alpha: load the MMU context when switch_mm() switches the current task Cc: # 01df8f401888: alpha: run check_mmu_context() from finish_arch_post_lock_switch() Cc: # ef43cfc28c42: alpha: only use a targeted tbi() when the target mm is really current Cc: # 738cf8f7a78f: alpha: invalidate the local context in flush_tlb_page() Cc: # 62c92086f30e: alpha: fix the local TLB invalidate in flush_tlb_page() Cc: # 3faee097b939: alpha: fix the local TLB invalidate in the UP flush_tlb_page() Cc: # b2c3b654bb64: alpha: invalidate the local context in flush_tlb_mm() Cc: # f19ae6f51a38: alpha: invalidate the local context in flush_icache_user_page() Cc: # 471e7f9c890b: alpha: do not clear remote MMU contexts in migrate_flush_tlb_page() Cc: stable@vger.kernel.org Reviewed-by: Matt Turner Signed-off-by: Magnus Lindholm Link: https://lore.kernel.org/r/20261010144621.974850-3-linmag7@gmail.com Signed-off-by: Magnus Lindholm Signed-off-by: Sasha Levin --- arch/alpha/include/asm/mmu_context.h | 7 +++- arch/alpha/kernel/smp.c | 52 ++++++++++++++-------------- arch/alpha/mm/fault.c | 4 ++- arch/alpha/mm/tlbflush.c | 2 +- 4 files changed, 36 insertions(+), 29 deletions(-) diff --git a/arch/alpha/include/asm/mmu_context.h b/arch/alpha/include/asm/mmu_context.h index f41d7c73fdd64..09f356934b90a 100644 --- a/arch/alpha/include/asm/mmu_context.h +++ b/arch/alpha/include/asm/mmu_context.h @@ -159,7 +159,12 @@ ev5_switch_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm, mmc = next_mm->context[cpu]; if ((mmc ^ asn) & ~HARDWARE_ASN_MASK) { mmc = __get_new_mm_context(next_mm, cpu); - next_mm->context[cpu] = mmc; + /* + * Ordered before any user access through the mm by the + * mb() in arch_spin_unlock(), when finish_lock_switch() + * drops the rq lock. + */ + WRITE_ONCE(next_mm->context[cpu], mmc); } #ifdef CONFIG_SMP /* A deferred shootdown can also invalidate a newly allocated ASN. */ diff --git a/arch/alpha/kernel/smp.c b/arch/alpha/kernel/smp.c index e21bc3920bece..11768aa292a48 100644 --- a/arch/alpha/kernel/smp.c +++ b/arch/alpha/kernel/smp.c @@ -625,12 +625,30 @@ static void ipi_flush_tlb_mm(void *x) { struct mm_struct *mm = x; - if (mm == current->active_mm && !asn_locked()) + if (mm == current->mm && !asn_locked()) flush_tlb_current(mm); else flush_tlb_other(mm); } +/* True if a CPU other than this one holds an ASN for MM. */ +static bool +mm_context_elsewhere(struct mm_struct *mm) +{ + int cpu, this_cpu = smp_processor_id(); + + /* Pairs with the barrier the publishing CPU issues before using MM. */ + smp_mb(); + + for_each_online_cpu(cpu) { + if (cpu == this_cpu) + continue; + if (READ_ONCE(mm->context[cpu])) + return true; + } + return false; +} + void flush_tlb_mm(struct mm_struct *mm) { @@ -638,14 +656,8 @@ flush_tlb_mm(struct mm_struct *mm) if (mm == current->active_mm) { flush_tlb_current(mm); - if (atomic_read(&mm->mm_users) <= 1) { - int cpu, this_cpu = smp_processor_id(); - for (cpu = 0; cpu < NR_CPUS; cpu++) { - if (!cpu_online(cpu) || cpu == this_cpu) - continue; - if (mm->context[cpu]) - mm->context[cpu] = 0; - } + if (atomic_read(&mm->mm_users) <= 1 && + !mm_context_elsewhere(mm)) { preempt_enable(); return; } @@ -690,14 +702,8 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) /* As in ipi_flush_tlb_page(): a targeted tbi() needs MM current. */ if (mm == current->mm) { flush_tlb_current_page(mm, vma, addr); - if (atomic_read(&mm->mm_users) <= 1) { - int cpu, this_cpu = smp_processor_id(); - for (cpu = 0; cpu < NR_CPUS; cpu++) { - if (!cpu_online(cpu) || cpu == this_cpu) - continue; - if (mm->context[cpu]) - mm->context[cpu] = 0; - } + if (atomic_read(&mm->mm_users) <= 1 && + !mm_context_elsewhere(mm)) { preempt_enable(); return; } @@ -728,7 +734,7 @@ static void ipi_flush_icache_page(void *x) { struct mm_struct *mm = (struct mm_struct *) x; - if (mm == current->active_mm && !asn_locked()) + if (mm == current->mm && !asn_locked()) __load_new_mm_context(mm); else flush_tlb_other(mm); @@ -747,14 +753,8 @@ flush_icache_user_page(struct vm_area_struct *vma, struct page *page, if (mm == current->active_mm) { __load_new_mm_context(mm); - if (atomic_read(&mm->mm_users) <= 1) { - int cpu, this_cpu = smp_processor_id(); - for (cpu = 0; cpu < NR_CPUS; cpu++) { - if (!cpu_online(cpu) || cpu == this_cpu) - continue; - if (mm->context[cpu]) - mm->context[cpu] = 0; - } + if (atomic_read(&mm->mm_users) <= 1 && + !mm_context_elsewhere(mm)) { preempt_enable(); return; } diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c index f40ac0826a854..53e170e965915 100644 --- a/arch/alpha/mm/fault.c +++ b/arch/alpha/mm/fault.c @@ -46,7 +46,9 @@ __load_new_mm_context(struct mm_struct *next_mm) struct pcb_struct *pcb; mmc = __get_new_mm_context(next_mm, smp_processor_id()); - next_mm->context[smp_processor_id()] = mmc; + WRITE_ONCE(next_mm->context[smp_processor_id()], mmc); + /* Publish the context before this CPU can access the mm. */ + smp_mb(); pcb = ¤t_thread_info()->pcb; pcb->asn = mmc & HARDWARE_ASN_MASK; diff --git a/arch/alpha/mm/tlbflush.c b/arch/alpha/mm/tlbflush.c index 239c72b8a7412..a2aef90e6234b 100644 --- a/arch/alpha/mm/tlbflush.c +++ b/arch/alpha/mm/tlbflush.c @@ -66,7 +66,7 @@ static void ipi_flush_mm_and_page(void *x) struct tlb_mm_and_addr *d = x; /* Part 1: mm context side (Alpha uses ASN/context as a key mechanism). */ - if (d->mm == current->active_mm && !asn_locked()) + if (d->mm == current->mm && !asn_locked()) __load_new_mm_context(d->mm); else flush_tlb_other(d->mm); -- 2.53.0