* [PATCH 1/2] powerpc/mm: Add tracking of the number of coprocessors using a context @ 2018-03-22 22:29 Balbir Singh 2018-03-22 22:29 ` [PATCH 2/2] powerpc/mm: Workaround Nest MMU bug with TLB invalidations Balbir Singh 2018-03-23 11:11 ` [1/2] powerpc/mm: Add tracking of the number of coprocessors using a context Michael Ellerman 0 siblings, 2 replies; 3+ messages in thread From: Balbir Singh @ 2018-03-22 22:29 UTC (permalink / raw) To: linuxppc-dev; +Cc: mpe, benh, npiggin, aneesh.kumar From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Currently, when using coprocessors (which use the Nest MMU), we simply increment the active_cpu count to force all TLB invalidations to be come broadcast. Unfortunately, due to an errata in POWER9, we will need to know more specifically that coprocessors are in use. This maintains a separate copros counter in the MMU context for that purpose. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Tested-by: Balbir Singh <bsingharora@gmail.com> --- arch/powerpc/include/asm/book3s/64/mmu.h | 3 +++ arch/powerpc/include/asm/mmu_context.h | 18 +++++++++++++----- arch/powerpc/mm/mmu_context_book3s64.c | 1 + 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h index bef6e39ed63a..729902e733cd 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu.h +++ b/arch/powerpc/include/asm/book3s/64/mmu.h @@ -87,6 +87,9 @@ typedef struct { /* Number of bits in the mm_cpumask */ atomic_t active_cpus; + /* Number of users of the external (Nest) MMU */ + atomic_t copros; + /* NPU NMMU context */ struct npu_context *npu_context; diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index 051b3d63afe3..3a15b6db9501 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -92,15 +92,23 @@ static inline void dec_mm_active_cpus(struct mm_struct *mm) static inline void mm_context_add_copro(struct mm_struct *mm) { /* - * On hash, should only be called once over the lifetime of - * the context, as we can't decrement the active cpus count - * and flush properly for the time being. + * If any copro is in use, increment the active CPU count + * in order to force TLB invalidations to be global as to + * propagate to the Nest MMU. */ - inc_mm_active_cpus(mm); + if (atomic_inc_return(&mm->context.copros) == 1) + inc_mm_active_cpus(mm); } static inline void mm_context_remove_copro(struct mm_struct *mm) { + int c; + + c = atomic_dec_if_positive(&mm->context.copros); + + /* Detect imbalance between add and remove */ + WARN_ON(c < 0); + /* * Need to broadcast a global flush of the full mm before * decrementing active_cpus count, as the next TLBI may be @@ -111,7 +119,7 @@ static inline void mm_context_remove_copro(struct mm_struct *mm) * for the time being. Invalidations will remain global if * used on hash. */ - if (radix_enabled()) { + if (c == 0 && radix_enabled()) { flush_all_mm(mm); dec_mm_active_cpus(mm); } diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c index 929d9ef7083f..3f980baade4c 100644 --- a/arch/powerpc/mm/mmu_context_book3s64.c +++ b/arch/powerpc/mm/mmu_context_book3s64.c @@ -173,6 +173,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm) mm_iommu_init(mm); #endif atomic_set(&mm->context.active_cpus, 0); + atomic_set(&mm->context.copros, 0); return 0; } -- 2.13.6 ^ permalink raw reply related [flat|nested] 3+ messages in thread
* [PATCH 2/2] powerpc/mm: Workaround Nest MMU bug with TLB invalidations 2018-03-22 22:29 [PATCH 1/2] powerpc/mm: Add tracking of the number of coprocessors using a context Balbir Singh @ 2018-03-22 22:29 ` Balbir Singh 2018-03-23 11:11 ` [1/2] powerpc/mm: Add tracking of the number of coprocessors using a context Michael Ellerman 1 sibling, 0 replies; 3+ messages in thread From: Balbir Singh @ 2018-03-22 22:29 UTC (permalink / raw) To: linuxppc-dev; +Cc: mpe, benh, npiggin, aneesh.kumar, Balbir Singh From: Benjamin Herrenschmidt <benh@kernel.crashing.org> On POWER9 the Nest MMU may fail to invalidate some translations when doing a tlbie "by PID" or "by LPID" that is targeted at the TLB only and not the page walk cache. This works around it by forcing such invalidations to escalate to RIC=2 (full invalidation of TLB *and* PWC) when a coprocessor is in use for the context. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Balbir Singh <bsingharora@gmail.com> [fixed a spelling and coding style to quiesce checkpatch.pl] Tested-by: Balbir Singh <bsingharora@gmail.com> --- arch/powerpc/mm/tlb-radix.c | 50 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 43 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 71d1b19ad1c0..8e4c6cb4a808 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -151,7 +151,23 @@ static inline void _tlbiel_pid(unsigned long pid, unsigned long ric) static inline void _tlbie_pid(unsigned long pid, unsigned long ric) { asm volatile("ptesync": : :"memory"); - __tlbie_pid(pid, ric); + + /* + * Workaround the fact that the "ric" argument to __tlbie_pid + * must be a compile-time contraint to match the "i" constraint + * in the asm statement. + */ + switch (ric) { + case RIC_FLUSH_TLB: + __tlbie_pid(pid, RIC_FLUSH_TLB); + break; + case RIC_FLUSH_PWC: + __tlbie_pid(pid, RIC_FLUSH_PWC); + break; + case RIC_FLUSH_ALL: + default: + __tlbie_pid(pid, RIC_FLUSH_ALL); + } asm volatile("eieio; tlbsync; ptesync": : :"memory"); } @@ -311,6 +327,16 @@ void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmadd } EXPORT_SYMBOL(radix__local_flush_tlb_page); +static bool mm_needs_flush_escalation(struct mm_struct *mm) +{ + /* + * P9 nest MMU has issues with the page walk cache + * caching PTEs and not flushing them properly when + * RIC = 0 for a PID/LPID invalidate + */ + return atomic_read(&mm->context.copros) != 0; +} + #ifdef CONFIG_SMP void radix__flush_tlb_mm(struct mm_struct *mm) { @@ -321,9 +347,12 @@ void radix__flush_tlb_mm(struct mm_struct *mm) return; preempt_disable(); - if (!mm_is_thread_local(mm)) - _tlbie_pid(pid, RIC_FLUSH_TLB); - else + if (!mm_is_thread_local(mm)) { + if (mm_needs_flush_escalation(mm)) + _tlbie_pid(pid, RIC_FLUSH_ALL); + else + _tlbie_pid(pid, RIC_FLUSH_TLB); + } else _tlbiel_pid(pid, RIC_FLUSH_TLB); preempt_enable(); } @@ -435,10 +464,14 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, } if (full) { - if (local) + if (local) { _tlbiel_pid(pid, RIC_FLUSH_TLB); - else - _tlbie_pid(pid, RIC_FLUSH_TLB); + } else { + if (mm_needs_flush_escalation(mm)) + _tlbie_pid(pid, RIC_FLUSH_ALL); + else + _tlbie_pid(pid, RIC_FLUSH_TLB); + } } else { bool hflush = false; unsigned long hstart, hend; @@ -548,6 +581,9 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, } if (full) { + if (!local && mm_needs_flush_escalation(mm)) + also_pwc = true; + if (local) _tlbiel_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); else -- 2.13.6 ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [1/2] powerpc/mm: Add tracking of the number of coprocessors using a context 2018-03-22 22:29 [PATCH 1/2] powerpc/mm: Add tracking of the number of coprocessors using a context Balbir Singh 2018-03-22 22:29 ` [PATCH 2/2] powerpc/mm: Workaround Nest MMU bug with TLB invalidations Balbir Singh @ 2018-03-23 11:11 ` Michael Ellerman 1 sibling, 0 replies; 3+ messages in thread From: Michael Ellerman @ 2018-03-23 11:11 UTC (permalink / raw) To: Balbir Singh, linuxppc-dev; +Cc: aneesh.kumar, npiggin On Thu, 2018-03-22 at 22:29:05 UTC, Balbir Singh wrote: > From: Benjamin Herrenschmidt <benh@kernel.crashing.org> > > Currently, when using coprocessors (which use the Nest MMU), we > simply increment the active_cpu count to force all TLB invalidations > to be come broadcast. > > Unfortunately, due to an errata in POWER9, we will need to know > more specifically that coprocessors are in use. > > This maintains a separate copros counter in the MMU context for > that purpose. > > Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> > Tested-by: Balbir Singh <bsingharora@gmail.com> Series applied to powerpc fixes, thanks. https://git.kernel.org/powerpc/c/aff6f8cb3e2170b9e58b0932bce7bf cheers ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2018-03-23 11:11 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-03-22 22:29 [PATCH 1/2] powerpc/mm: Add tracking of the number of coprocessors using a context Balbir Singh 2018-03-22 22:29 ` [PATCH 2/2] powerpc/mm: Workaround Nest MMU bug with TLB invalidations Balbir Singh 2018-03-23 11:11 ` [1/2] powerpc/mm: Add tracking of the number of coprocessors using a context Michael Ellerman
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).