From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98A05C64ED6 for ; Sun, 26 Feb 2023 22:13:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229601AbjBZWNe (ORCPT ); Sun, 26 Feb 2023 17:13:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229530AbjBZWNe (ORCPT ); Sun, 26 Feb 2023 17:13:34 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A1BDD31A for ; Sun, 26 Feb 2023 14:13:32 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 02927B80C9A for ; Sun, 26 Feb 2023 22:13:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A9B23C433D2; Sun, 26 Feb 2023 22:13:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1677449609; bh=l6IhCOcsrxF31GCeHL2YIQ5iMyNKpM14wXKmDJd/q8k=; h=Date:To:From:Subject:From; b=SWq3RBRZYWy3O1UvHDf9oqwCe3/ASZMynqMhrIN9c09A0F6eGTmqp5TuI2KiMwoqB rE2dfHSdsHNWV/KMD83lLsSz+pNnTZVWPDqfYSCLjIvAWGwwSrzJKtfDUM3QjPr2Mv H69nAUCWN3M4xthFB4EvXo4vk59UqovDPkmmfLN4= Date: Sun, 26 Feb 2023 14:13:29 -0800 To: mm-commits@vger.kernel.org, will@kernel.org, torvalds@linux-foundation.org, riel@redhat.com, peterz@infradead.org, nadav.amit@gmail.com, mpe@ellerman.id.au, luto@kernel.org, dave.hansen@linux.intel.com, christophe.leroy@csgroup.eu, catalin.marinas@arm.com, npiggin@gmail.com, akpm@linux-foundation.org From: Andrew Morton Subject: + lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions.patch added to mm-unstable branch Message-Id: <20230226221329.A9B23C433D2@smtp.kernel.org> Precedence: bulk Reply-To: linux-kernel@vger.kernel.org List-ID: X-Mailing-List: mm-commits@vger.kernel.org The patch titled Subject: lazy tlb: introduce lazy tlb mm refcount helper functions has been added to the -mm mm-unstable branch. Its filename is lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions.patch This patch will later appear in the mm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: Nicholas Piggin Subject: lazy tlb: introduce lazy tlb mm refcount helper functions Date: Fri, 3 Feb 2023 17:18:34 +1000 Add explicit _lazy_tlb annotated functions for lazy tlb mm refcounting. This makes the lazy tlb mm references more obvious, and allows the refcounting scheme to be modified in later changes. There is no functional change with this patch. Link: https://lkml.kernel.org/r/20230203071837.1136453-3-npiggin@gmail.com Signed-off-by: Nicholas Piggin Acked-by: Linus Torvalds Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Christophe Leroy Cc: Dave Hansen Cc: Michael Ellerman Cc: Nadav Amit Cc: Peter Zijlstra Cc: Rik van Riel Cc: Will Deacon Signed-off-by: Andrew Morton --- diff --git a/arch/arm/mach-rpc/ecard.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/arch/arm/mach-rpc/ecard.c --- a/arch/arm/mach-rpc/ecard.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/arch/arm/mach-rpc/ecard.c @@ -253,7 +253,7 @@ static int ecard_init_mm(void) current->mm = mm; current->active_mm = mm; activate_mm(active_mm, mm); - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); ecard_init_pgtables(mm); return 0; } diff --git a/arch/powerpc/kernel/smp.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/arch/powerpc/kernel/smp.c --- a/arch/powerpc/kernel/smp.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/arch/powerpc/kernel/smp.c @@ -1611,7 +1611,7 @@ void start_secondary(void *unused) if (IS_ENABLED(CONFIG_PPC32)) setup_kup(); - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; smp_store_cpu_info(cpu); diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/arch/powerpc/mm/book3s64/radix_tlb.c --- a/arch/powerpc/mm/book3s64/radix_tlb.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -797,10 +797,10 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush) if (current->active_mm == mm) { WARN_ON_ONCE(current->mm != NULL); /* Is a kernel thread and is using mm as the lazy tlb */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; switch_mm_irqs_off(mm, &init_mm, current); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } /* diff --git a/fs/exec.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/fs/exec.c --- a/fs/exec.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/fs/exec.c @@ -1034,7 +1034,7 @@ static int exec_mmap(struct mm_struct *mm) mmput(old_mm); return 0; } - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); return 0; } diff --git a/include/linux/sched/mm.h~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/include/linux/sched/mm.h --- a/include/linux/sched/mm.h~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/include/linux/sched/mm.h @@ -79,6 +79,22 @@ static inline void mmdrop_sched(struct mm_struct *mm) } #endif +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb(struct mm_struct *mm) +{ + mmgrab(mm); +} + +static inline void mmdrop_lazy_tlb(struct mm_struct *mm) +{ + mmdrop(mm); +} + +static inline void mmdrop_lazy_tlb_sched(struct mm_struct *mm) +{ + mmdrop_sched(mm); +} + /** * mmget() - Pin the address space associated with a &struct mm_struct. * @mm: The address space to pin. diff --git a/kernel/cpu.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/kernel/cpu.c --- a/kernel/cpu.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/kernel/cpu.c @@ -623,7 +623,7 @@ static int finish_cpu(unsigned int cpu) */ if (mm != &init_mm) idle->active_mm = &init_mm; - mmdrop(mm); + mmdrop_lazy_tlb(mm); return 0; } diff --git a/kernel/exit.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/kernel/exit.c --- a/kernel/exit.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/kernel/exit.c @@ -537,7 +537,7 @@ static void exit_mm(void) return; sync_mm_rss(mm); mmap_read_lock(mm); - mmgrab(mm); + mmgrab_lazy_tlb(mm); BUG_ON(mm != current->active_mm); /* more a memory barrier than a real lock */ task_lock(current); diff --git a/kernel/kthread.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/kernel/kthread.c --- a/kernel/kthread.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/kernel/kthread.c @@ -1415,6 +1415,11 @@ void kthread_use_mm(struct mm_struct *mm) WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)); WARN_ON_ONCE(tsk->mm); + /* + * It is possible for mm to be the same as tsk->active_mm, but + * we must still mmgrab(mm) and mmdrop_lazy_tlb(active_mm), + * because these references are not equivalent. + */ mmgrab(mm); task_lock(tsk); @@ -1438,9 +1443,9 @@ void kthread_use_mm(struct mm_struct *mm) * memory barrier after storing to tsk->mm, before accessing * user-space memory. A full memory barrier for membarrier * {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by - * mmdrop(). + * mmdrop_lazy_tlb(). */ - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); } EXPORT_SYMBOL_GPL(kthread_use_mm); @@ -1468,10 +1473,13 @@ void kthread_unuse_mm(struct mm_struct *mm) local_irq_disable(); tsk->mm = NULL; membarrier_update_current_mm(NULL); + mmgrab_lazy_tlb(mm); /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); local_irq_enable(); task_unlock(tsk); + + mmdrop(mm); } EXPORT_SYMBOL_GPL(kthread_unuse_mm); diff --git a/kernel/sched/core.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions b/kernel/sched/core.c --- a/kernel/sched/core.c~lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions +++ b/kernel/sched/core.c @@ -5200,13 +5200,14 @@ static struct rq *finish_task_switch(struct task_struct *prev) * rq->curr, before returning to userspace, so provide them here: * * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly - * provided by mmdrop(), + * provided by mmdrop_lazy_tlb(), * - a sync_core for SYNC_CORE. */ if (mm) { membarrier_mm_sync_core_before_usermode(mm); - mmdrop_sched(mm); + mmdrop_lazy_tlb_sched(mm); } + if (unlikely(prev_state == TASK_DEAD)) { if (prev->sched_class->task_dead) prev->sched_class->task_dead(prev); @@ -5263,9 +5264,9 @@ context_switch(struct rq *rq, struct task_struct *prev, /* * kernel -> kernel lazy + transfer active - * user -> kernel lazy + mmgrab() active + * user -> kernel lazy + mmgrab_lazy_tlb() active * - * kernel -> user switch + mmdrop() active + * kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) { // to kernel @@ -5273,7 +5274,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm; if (prev->mm) // from user - mmgrab(prev->active_mm); + mmgrab_lazy_tlb(prev->active_mm); else prev->active_mm = NULL; } else { // to user @@ -5290,7 +5291,7 @@ context_switch(struct rq *rq, struct task_struct *prev, lru_gen_use_mm(next->mm); if (!prev->mm) { // from kernel - /* will mmdrop() in finish_task_switch(). */ + /* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm = NULL; } @@ -9932,7 +9933,7 @@ void __init sched_init(void) /* * The boot idle thread does lazy MMU switching as well: */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); enter_lazy_tlb(&init_mm, current); /* _ Patches currently in -mm which might be from npiggin@gmail.com are kthread-simplify-kthread_use_mm-refcounting.patch lazy-tlb-introduce-lazy-tlb-mm-refcount-helper-functions.patch lazy-tlb-allow-lazy-tlb-mm-refcounting-to-be-configurable.patch lazy-tlb-shoot-lazies-non-refcounting-lazy-tlb-mm-reference-handling-scheme.patch powerpc-64s-enable-mmu_lazy_tlb_shootdown.patch