From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Zijlstra Subject: Re: [PATCH 08/20] mm: Optimize fullmm TLB flushing Date: Thu, 28 Jun 2012 18:20:25 +0200 Message-ID: <1340900425.28750.73.camel@twins> References: <20120627211540.459910855@chello.nl> <20120627212831.137126018@chello.nl> <1340838154.10063.86.camel@twins> <1340838807.10063.90.camel@twins> <20120628091627.GB8573@arm.com> <1340879984.20977.80.camel@pasglop> <1340881196.28750.16.camel@twins> <20120628145327.GA17242@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20120628145327.GA17242@arm.com> Sender: owner-linux-mm@kvack.org To: Catalin Marinas Cc: Benjamin Herrenschmidt , Linus Torvalds , "linux-kernel@vger.kernel.org" , "linux-arch@vger.kernel.org" , "linux-mm@kvack.org" , Thomas Gleixner , Ingo Molnar , "akpm@linux-foundation.org" , Rik van Riel , Hugh Dickins , Mel Gorman , Nick Piggin , Alex Shi , "Nikunj A. Dadhania" , Konrad Rzeszutek Wilk , David Miller , Russell King , Chris Metcalf , Martin Schwidefsky , Tony Luck , Paul Mundt , Jeff List-Id: linux-arch.vger.kernel.org On Thu, 2012-06-28 at 15:53 +0100, Catalin Marinas wrote: > > Yes they do.. its just the up-front TLB invalidate for fullmm that's a > > problem. >=20 > The upfront invalidate is fine (i.e. harmless), it's the tlb_flush_mmu() > change to check for !tlb->fullmm that's not helpful on ARM. I think we're saying the same but differently. The point is that the flush up front isn't sufficient for most of us. Also, we'd very much want to avoid superfluous flushes since they are somewhat expensive. How horrid is something like the below. It detaches the mm so that hardware speculation simply doesn't matter. Now the switch_mm should imply the same cache+TBL flush we'd otherwise do, and I'd think that that would be the majority of the cost. Am I wrong there? Also, the below seems to leak mm_structs so I did mess up the ref-counting, its too bloody hot here. --- mm/memory.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 47 insertions(+), 4 deletions(-) --- a/mm/memory.c +++ b/mm/memory.c @@ -65,6 +65,7 @@ #include #include #include +#include =20 #include "internal.h" =20 @@ -197,6 +198,33 @@ static int tlb_next_batch(struct mmu_gat return 1; } =20 +/* + * Anonymize the task by detaching the mm and attaching it + * to the init_mm. + */ +static void detach_mm(struct mm_struct *mm, struct task_struct *tsk) +{ + /* + * We should only be called when there's no users left and we're + * destroying the mm. + */ + VM_BUG_ON(atomic_read(&mm->mm_users)); + VM_BUG_ON(tsk->mm !=3D mm); + VM_BUG_ON(mm =3D=3D &init_mm); + + task_lock(tsk); + tsk->mm =3D NULL; + tsk->active_mm =3D &init_mm; + switch_mm(mm, &init_mm, tsk); + /* + * We have to take an extra ref on init_mm for TASK_DEAD in + * finish_task_switch(), we don't drop our mm->mm_count reference + * since mmput() will do this. + */ + atomic_inc(&init_mm.mm_count); + task_unlock(tsk); +} + /* tlb_gather_mmu * Called to initialize an (on-stack) mmu_gather structure for page-table * tear-down from @mm. The @fullmm argument is used when @mm is without @@ -215,16 +243,31 @@ void tlb_gather_mmu(struct mmu_gather *t tlb->active =3D &tlb->local; =20 tlb_table_init(tlb); + + if (fullmm && current->mm =3D=3D mm) { + /* + * Instead of doing: + * + * flush_cache_mm(mm); + * flush_tlb_mm(mm); + * + * We switch to init_mm, this context switch should imply both + * the cache and TLB flush as well as guarantee that hardware + * speculation cannot load TLBs on this mm anymore. + */ + detach_mm(mm, current); + } } =20 void tlb_flush_mmu(struct mmu_gather *tlb) { struct mmu_gather_batch *batch; =20 - if (!tlb->need_flush) - return; - tlb->need_flush =3D 0; - flush_tlb_mm(tlb->mm); + if (!tlb->fullmm && tlb->need_flush) { + tlb->need_flush =3D 0; + flush_tlb_mm(tlb->mm); + } + tlb_table_flush(tlb); =20 if (tlb_fast_mode(tlb)) -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org