* [PATCH v2 19/22] powerpc/kvm/book3s: Use pte_present instead of opencoding _PAGE_PRESENT check
From: Aneesh Kumar K.V @ 2020-03-19 3:56 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200319035609.158654-1-aneesh.kumar@linux.ibm.com>
This adds _PAGE_PTE check and makes sure we validate the pte value returned via
find_kvm_host_pte.
NOTE: this also considers _PAGE_INVALID to the software valid bit.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
index 1ca1f6495012..c58e64a0a74f 100644
--- a/arch/powerpc/include/asm/kvm_book3s_64.h
+++ b/arch/powerpc/include/asm/kvm_book3s_64.h
@@ -435,7 +435,7 @@ static inline pte_t kvmppc_read_update_linux_pte(pte_t *ptep, int writing)
continue;
}
/* If pte is not present return None */
- if (unlikely(!(pte_val(old_pte) & _PAGE_PRESENT)))
+ if (unlikely(!pte_present(old_pte)))
return __pte(0);
new_pte = pte_mkyoung(old_pte);
--
2.24.1
^ permalink raw reply related
* [PATCH v2 20/22] powerpc/mm/book3s64: Avoid sending IPI on clearing PMD
From: Aneesh Kumar K.V @ 2020-03-19 3:56 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200319035609.158654-1-aneesh.kumar@linux.ibm.com>
Now that all the lockless page table walk is careful w.r.t the PTE
address returned, we can now revert
commit: 13bd817bb884 ("powerpc/thp: Serialize pmd clear against a linux page table walk.")
We also drop the equivalent IPI from other pte updates routines. We still keep
IPI in hash pmdp collapse and that is to take care of parallel hash page table
insert. The radix pmdp collapse flush can possibly be removed once I am sure
generic code doesn't have the any expectations around parallel gup walk.
This speeds up Qemu guest RAM del/unplug time as below
128 core, 496GB guest:
Without patch:
munmap start: timer = 13162 ms, PID=7684
munmap finish: timer = 95312 ms, PID=7684 - delta = 82150 ms
With patch:
munmap start: timer = 196449 ms, PID=6681
munmap finish: timer = 196488 ms, PID=6681 - delta = 39ms
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/mm/book3s64/hash_pgtable.c | 11 -----------
arch/powerpc/mm/book3s64/pgtable.c | 8 --------
arch/powerpc/mm/book3s64/radix_pgtable.c | 19 +++++++------------
3 files changed, 7 insertions(+), 31 deletions(-)
diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c
index 64733b9cb20a..64ca375278dc 100644
--- a/arch/powerpc/mm/book3s64/hash_pgtable.c
+++ b/arch/powerpc/mm/book3s64/hash_pgtable.c
@@ -363,17 +363,6 @@ pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm,
* hash fault look at them.
*/
memset(pgtable, 0, PTE_FRAG_SIZE);
- /*
- * Serialize against find_current_mm_pte variants which does lock-less
- * lookup in page tables with local interrupts disabled. For huge pages
- * it casts pmd_t to pte_t. Since format of pte_t is different from
- * pmd_t we want to prevent transit from pmd pointing to page table
- * to pmd pointing to huge page (and back) while interrupts are disabled.
- * We clear pmd to possibly replace it with page table pointer in
- * different code paths. So make sure we wait for the parallel
- * find_curren_mm_pte to finish.
- */
- serialize_against_pte_lookup(mm);
return old_pmd;
}
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index 2bf7e1b4fd82..93fc3be41ed9 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -109,14 +109,6 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
old_pmd = pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, _PAGE_INVALID);
flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
- /*
- * This ensures that generic code that rely on IRQ disabling
- * to prevent a parallel THP split work as expected.
- *
- * Marking the entry with _PAGE_INVALID && ~_PAGE_PRESENT requires
- * a special case check in pmd_access_permitted.
- */
- serialize_against_pte_lookup(vma->vm_mm);
return __pmd(old_pmd);
}
diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
index dd1bea45325c..0a78b10c256c 100644
--- a/arch/powerpc/mm/book3s64/radix_pgtable.c
+++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
@@ -957,7 +957,13 @@ pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long addre
pmd = *pmdp;
pmd_clear(pmdp);
- /*FIXME!! Verify whether we need this kick below */
+ /*
+ * pmdp collapse_flush need to ensure that there are no parallel gup
+ * walk after this call. This is needed so that we can have stable
+ * page ref count when collapsing a page. We don't allow a collapse page
+ * if we have gup taken on the page. We can ensure that by sending IPI
+ * because gup walk happens with IRQ disabled.
+ */
serialize_against_pte_lookup(vma->vm_mm);
radix__flush_tlb_collapsed_pmd(vma->vm_mm, address);
@@ -1018,17 +1024,6 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,
old = radix__pmd_hugepage_update(mm, addr, pmdp, ~0UL, 0);
old_pmd = __pmd(old);
- /*
- * Serialize against find_current_mm_pte which does lock-less
- * lookup in page tables with local interrupts disabled. For huge pages
- * it casts pmd_t to pte_t. Since format of pte_t is different from
- * pmd_t we want to prevent transit from pmd pointing to page table
- * to pmd pointing to huge page (and back) while interrupts are disabled.
- * We clear pmd to possibly replace it with page table pointer in
- * different code paths. So make sure we wait for the parallel
- * find_current_mm_pte to finish.
- */
- serialize_against_pte_lookup(mm);
return old_pmd;
}
--
2.24.1
^ permalink raw reply related
* [PATCH v2 21/22] mm: change pmdp_huge_get_and_clear_full take vm_area_struct as arg
From: Aneesh Kumar K.V @ 2020-03-19 3:56 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200319035609.158654-1-aneesh.kumar@linux.ibm.com>
We will use this in later patch to do tlb flush when clearing pmd entries.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/s390/include/asm/pgtable.h | 4 ++--
include/asm-generic/pgtable.h | 4 ++--
mm/huge_memory.c | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 6d7c3b7e9281..4efee8025c97 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -1535,7 +1535,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
}
#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
-static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
+static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long addr,
pmd_t *pmdp, int full)
{
@@ -1544,7 +1544,7 @@ static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
*pmdp = __pmd(_SEGMENT_ENTRY_EMPTY);
return pmd;
}
- return pmdp_xchg_lazy(mm, addr, pmdp, __pmd(_SEGMENT_ENTRY_EMPTY));
+ return pmdp_xchg_lazy(vma->vm_mm, addr, pmdp, __pmd(_SEGMENT_ENTRY_EMPTY));
}
#define __HAVE_ARCH_PMDP_HUGE_CLEAR_FLUSH
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index e2e2bef07dd2..bb0c7dc0a3d9 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -158,11 +158,11 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
-static inline pmd_t pmdp_huge_get_and_clear_full(struct mm_struct *mm,
+static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long address, pmd_t *pmdp,
int full)
{
- return pmdp_huge_get_and_clear(mm, address, pmdp);
+ return pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
}
#endif
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 24ad53b4dfc0..a5e0825cc00b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1799,8 +1799,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
* pgtable_trans_huge_withdraw after finishing pmdp related
* operations.
*/
- orig_pmd = pmdp_huge_get_and_clear_full(tlb->mm, addr, pmd,
- tlb->fullmm);
+ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd,
+ tlb->fullmm);
tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
if (vma_is_dax(vma)) {
if (arch_needs_pgtable_deposit())
--
2.24.1
^ permalink raw reply related
* [PATCH v2 22/22] powerpc/mm/book3s64: Fix MADV_DONTNEED and parallel page fault race
From: Aneesh Kumar K.V @ 2020-03-19 3:56 UTC (permalink / raw)
To: linuxppc-dev, mpe, linux-mm, linux-kernel, kvm-ppc
Cc: leonardo, Aneesh Kumar K.V, npiggin, kirill
In-Reply-To: <20200319035609.158654-1-aneesh.kumar@linux.ibm.com>
MADV_DONTNEED holds mmap_sem in read mode and that implies a
parallel page fault is possible and the kernel can end up with a level 1 PTE
entry (THP entry) converted to a level 0 PTE entry without flushing
the THP TLB entry.
Most architectures including POWER have issues with kernel instantiating a level
0 PTE entry while holding level 1 TLB entries.
The code sequence I am looking at is
down_read(mmap_sem) down_read(mmap_sem)
zap_pmd_range()
zap_huge_pmd()
pmd lock held
pmd_cleared
table details added to mmu_gather
pmd_unlock()
insert a level 0 PTE entry()
tlb_finish_mmu().
Fix this by forcing a tlb flush before releasing pmd lock if this is
not a fullmm invalidate. We can safely skip this invalidate for
task exit case (fullmm invalidate) because in that case we are sure
there can be no parallel fault handlers.
This do change the Qemu guest RAM del/unplug time as below
128 core, 496GB guest:
Without patch:
munmap start: timer = 196449 ms, PID=6681
munmap finish: timer = 196488 ms, PID=6681 - delta = 39ms
With patch:
munmap start: timer = 196345 ms, PID=6879
munmap finish: timer = 196714 ms, PID=6879 - delta = 369ms
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
---
arch/powerpc/include/asm/book3s/64/pgtable.h | 5 +++++
arch/powerpc/mm/book3s64/pgtable.c | 18 ++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index 89eb7b350df8..e98cb141d490 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -1265,6 +1265,11 @@ static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
}
#define pmdp_collapse_flush pmdp_collapse_flush
+#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL
+pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
+ unsigned long addr,
+ pmd_t *pmdp, int full);
+
#define __HAVE_ARCH_PGTABLE_DEPOSIT
static inline void pgtable_trans_huge_deposit(struct mm_struct *mm,
pmd_t *pmdp, pgtable_t pgtable)
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index 93fc3be41ed9..fefdc7d41040 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -112,6 +112,24 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
return __pmd(old_pmd);
}
+pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmdp, int full)
+{
+ pmd_t pmd;
+ VM_BUG_ON(addr & ~HPAGE_PMD_MASK);
+ VM_BUG_ON((pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) &&
+ !pmd_devmap(*pmdp)) || !pmd_present(*pmdp));
+ pmd = pmdp_huge_get_and_clear(vma->vm_mm, addr, pmdp);
+ /*
+ * if it not a fullmm flush, then we can possibly end up converting
+ * this PMD pte entry to a regular level 0 PTE by a parallel page fault.
+ * Make sure we flush the tlb in this case.
+ */
+ if (!full)
+ flush_pmd_tlb_range(vma, addr, addr + HPAGE_PMD_SIZE);
+ return pmd;
+}
+
static pmd_t pmd_set_protbits(pmd_t pmd, pgprot_t pgprot)
{
return __pmd(pmd_val(pmd) | pgprot_val(pgprot));
--
2.24.1
^ permalink raw reply related
* Re: [PATCH v5 2/2] powerpc/64: Prevent stack protection in early boot
From: Michael Ellerman @ 2020-03-19 4:00 UTC (permalink / raw)
To: Daniel Axtens, linuxppc-dev; +Cc: npiggin
In-Reply-To: <87wo7hn8az.fsf@dja-thinkpad.axtens.net>
Daniel Axtens <dja@axtens.net> writes:
> Michael Ellerman <mpe@ellerman.id.au> writes:
>
>> The previous commit reduced the amount of code that is run before we
>> setup a paca. However there are still a few remaining functions that
>> run with no paca, or worse, with an arbitrary value in r13 that will
>> be used as a paca pointer.
>>
>> In particular the stack protector canary is stored in the paca, so if
>> stack protector is activated for any of these functions we will read
>> the stack canary from wherever r13 points. If r13 happens to point
>> outside of memory we will get a machine check / checkstop.
>>
>> For example if we modify initialise_paca() to trigger stack
>> protection, and then boot in the mambo simulator with r13 poisoned in
>> skiboot before calling the kernel:
>>
>> DEBUG: 19952232: (19952232): INSTRUCTION: PC=0xC0000000191FC1E8: [0x3C4C006D]: addis r2,r12,0x6D [fetch]
>> DEBUG: 19952236: (19952236): INSTRUCTION: PC=0xC00000001807EAD8: [0x7D8802A6]: mflr r12 [fetch]
>> FATAL ERROR: 19952276: (19952276): Check Stop for 0:0: Machine Check with ME bit of MSR off
>> DEBUG: 19952276: (19952276): INSTRUCTION: PC=0xC0000000191FCA7C: [0xE90D0CF8]: ld r8,0xCF8(r13) [Instruction Failed]
>> INFO: 19952276: (19952277): ** Execution stopped: Mambo Error, Machine Check Stop, **
>> systemsim % bt
>> pc: 0xC0000000191FCA7C initialise_paca+0x54
>> lr: 0xC0000000191FC22C early_setup+0x44
>> stack:0x00000000198CBED0 0x0 +0x0
>> stack:0x00000000198CBF00 0xC0000000191FC22C early_setup+0x44
>> stack:0x00000000198CBF90 0x1801C968 +0x1801C968
>>
>> So annotate the relevant functions to ensure stack protection is never
>> enabled for them.
>
> This all makes sense to me, although I don't really understand the stack
> protector especially well.
The key details for this bug are that 1) some functions get stack
protection, if they have on-stack buffers etc. 2) that stack protection
involves reading a canary from the paca.
> I have checked and I can find no other C functions that are called
> before early_setup.
Thanks.
Except for all of prom_init.c but that's already built with no stack
protector.
> Do we need to do add setup_64.c to the part of the Makefile that
> disables tracing of early boot?
>
> ifdef CONFIG_FUNCTION_TRACER
> # Do not trace early boot code
> CFLAGS_REMOVE_cputable.o = $(CC_FLAGS_FTRACE)
> CFLAGS_REMOVE_prom_init.o = $(CC_FLAGS_FTRACE)
> CFLAGS_REMOVE_btext.o = $(CC_FLAGS_FTRACE)
> CFLAGS_REMOVE_prom.o = $(CC_FLAGS_FTRACE)
> -> should we add setup_64.c here?
> endif
No I don't think so.
Tracing is less of a concern during very early boot because although the
functions may be built to support tracing, you can't actually turn
tracing *on* that early.
Also setup_64.c is not purely early boot code, there are some functions
in there we would like to be able to trace.
As I was saying the other day, we may want to create a specific
directory (or file) for all the really early boot code where we turn off
all special options like tracing, kcov, stack protector etc.
cheers
^ permalink raw reply
* Re: [PATCHv2 26/50] powerpc: Add show_stack_loglvl()
From: Michael Ellerman @ 2020-03-19 4:03 UTC (permalink / raw)
To: Dmitry Safonov, linux-kernel
Cc: Petr Mladek, Dmitry Safonov, Tetsuo Handa, Greg Kroah-Hartman,
Dmitry Safonov, Steven Rostedt, Sergey Senozhatsky,
Paul Mackerras, Jiri Slaby, Andrew Morton, linuxppc-dev,
Ingo Molnar
In-Reply-To: <20200316143916.195608-27-dima@arista.com>
Dmitry Safonov <dima@arista.com> writes:
> Currently, the log-level of show_stack() depends on a platform
> realization. It creates situations where the headers are printed with
> lower log level or higher than the stacktrace (depending on
> a platform or user).
>
> Furthermore, it forces the logic decision from user to an architecture
> side. In result, some users as sysrq/kdb/etc are doing tricks with
> temporary rising console_loglevel while printing their messages.
> And in result it not only may print unwanted messages from other CPUs,
> but also omit printing at all in the unlucky case where the printk()
> was deferred.
>
> Introducing log-level parameter and KERN_UNSUPPRESSED [1] seems
> an easier approach than introducing more printk buffers.
> Also, it will consolidate printings with headers.
>
> Introduce show_stack_loglvl(), that eventually will substitute
> show_stack().
>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: linuxppc-dev@lists.ozlabs.org
> [1]: https://lore.kernel.org/lkml/20190528002412.1625-1-dima@arista.com/T/#u
> Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
> Signed-off-by: Dmitry Safonov <dima@arista.com>
> ---
> arch/powerpc/kernel/process.c | 18 +++++++++++++-----
> 1 file changed, 13 insertions(+), 5 deletions(-)
Thanks for doing this, it has caused me problems in the past.
Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
cheers
> diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
> index fad50db9dcf2..c1ab7f613da4 100644
> --- a/arch/powerpc/kernel/process.c
> +++ b/arch/powerpc/kernel/process.c
> @@ -2034,7 +2034,8 @@ unsigned long get_wchan(struct task_struct *p)
>
> static int kstack_depth_to_print = CONFIG_PRINT_STACK_DEPTH;
>
> -void show_stack(struct task_struct *tsk, unsigned long *stack)
> +void show_stack_loglvl(struct task_struct *tsk, unsigned long *stack,
> + const char *loglvl)
> {
> unsigned long sp, ip, lr, newsp;
> int count = 0;
> @@ -2059,7 +2060,7 @@ void show_stack(struct task_struct *tsk, unsigned long *stack)
> }
>
> lr = 0;
> - printk("Call Trace:\n");
> + printk("%sCall Trace:\n", loglvl);
> do {
> if (!validate_sp(sp, tsk, STACK_FRAME_OVERHEAD))
> break;
> @@ -2068,7 +2069,8 @@ void show_stack(struct task_struct *tsk, unsigned long *stack)
> newsp = stack[0];
> ip = stack[STACK_FRAME_LR_SAVE];
> if (!firstframe || ip != lr) {
> - printk("["REG"] ["REG"] %pS", sp, ip, (void *)ip);
> + printk("%s["REG"] ["REG"] %pS",
> + loglvl, sp, ip, (void *)ip);
> #ifdef CONFIG_FUNCTION_GRAPH_TRACER
> ret_addr = ftrace_graph_ret_addr(current,
> &ftrace_idx, ip, stack);
> @@ -2090,8 +2092,9 @@ void show_stack(struct task_struct *tsk, unsigned long *stack)
> struct pt_regs *regs = (struct pt_regs *)
> (sp + STACK_FRAME_OVERHEAD);
> lr = regs->link;
> - printk("--- interrupt: %lx at %pS\n LR = %pS\n",
> - regs->trap, (void *)regs->nip, (void *)lr);
> + printk("%s--- interrupt: %lx at %pS\n LR = %pS\n",
> + loglvl, regs->trap,
> + (void *)regs->nip, (void *)lr);
> firstframe = 1;
> }
>
> @@ -2101,6 +2104,11 @@ void show_stack(struct task_struct *tsk, unsigned long *stack)
> put_task_stack(tsk);
> }
>
> +void show_stack(struct task_struct *tsk, unsigned long *stack)
> +{
> + show_stack_loglvl(tsk, stack, KERN_DEFAULT);
> +}
> +
> #ifdef CONFIG_PPC64
> /* Called with hard IRQs off */
> void notrace __ppc64_runlatch_on(void)
> --
> 2.25.1
^ permalink raw reply
* Re: [RFC PATCH] powerpc/64s: CONFIG_PPC_HASH_MMU
From: Michael Ellerman @ 2020-03-19 4:15 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20200302044902.3418935-1-npiggin@gmail.com>
Nicholas Piggin <npiggin@gmail.com> writes:
> This allows the 64s hash MMU code to be compiled out if radix is
> selected. This saves about 128kB kernel image size (90kB text) on
> powernv_defconfig minus KVM, 40kB on a tiny config.
TBH my feelings are:
- the size savings don't excite me much, given our kernels can be ~32MB
in size.
- it's unlikely a major (or any) distro would ever enable this, so it
would not be well tested.
- it adds a *lot* of #ifdefs.
So if we want to get this merged I'd want a good amount of refactoring
done to avoid the bulk of the #ifdefs in the code. ie. lots of wrapper
inlines etc. to hide the #ifdefs.
And someone to put up their hand to maintain it, ie. ensure it keeps
building and fix any issues that come up when it breaks.
cheers
> arch/powerpc/Kconfig | 1 +
> arch/powerpc/include/asm/book3s/64/mmu.h | 20 ++++++++++++++++-
> arch/powerpc/include/asm/book3s/64/pgtable.h | 6 +++++
> .../include/asm/book3s/64/tlbflush-hash.h | 10 +++++++--
> arch/powerpc/include/asm/book3s/64/tlbflush.h | 4 ----
> arch/powerpc/include/asm/book3s/pgtable.h | 4 ++++
> arch/powerpc/include/asm/mmu.h | 14 ++++++++++--
> arch/powerpc/include/asm/mmu_context.h | 2 ++
> arch/powerpc/include/asm/paca.h | 9 ++++++++
> arch/powerpc/include/asm/sparsemem.h | 2 +-
> arch/powerpc/kernel/asm-offsets.c | 4 ++++
> arch/powerpc/kernel/dt_cpu_ftrs.c | 10 ++++++++-
> arch/powerpc/kernel/entry_64.S | 4 ++--
> arch/powerpc/kernel/exceptions-64s.S | 22 ++++++++++++++++---
> arch/powerpc/kernel/mce.c | 2 +-
> arch/powerpc/kernel/mce_power.c | 10 ++++++---
> arch/powerpc/kernel/paca.c | 18 ++++++---------
> arch/powerpc/kernel/process.c | 13 ++++++-----
> arch/powerpc/kernel/prom.c | 2 ++
> arch/powerpc/kernel/setup_64.c | 4 ++++
> arch/powerpc/kexec/core_64.c | 4 ++--
> arch/powerpc/kvm/Kconfig | 1 +
> arch/powerpc/mm/book3s64/Makefile | 17 ++++++++------
> arch/powerpc/mm/book3s64/hash_pgtable.c | 1 -
> arch/powerpc/mm/book3s64/hash_utils.c | 9 --------
> .../{hash_hugetlbpage.c => hugetlbpage.c} | 2 ++
> arch/powerpc/mm/book3s64/mmu_context.c | 18 +++++++++++++--
> arch/powerpc/mm/book3s64/pgtable.c | 22 +++++++++++++++++--
> arch/powerpc/mm/book3s64/radix_pgtable.c | 5 +++++
> arch/powerpc/mm/book3s64/slb.c | 14 ------------
> arch/powerpc/mm/copro_fault.c | 2 ++
> arch/powerpc/mm/fault.c | 20 +++++++++++++++++
> arch/powerpc/platforms/Kconfig.cputype | 20 ++++++++++++++++-
> arch/powerpc/platforms/powernv/idle.c | 2 ++
> arch/powerpc/platforms/powernv/setup.c | 2 ++
> arch/powerpc/xmon/xmon.c | 8 +++++--
> 36 files changed, 231 insertions(+), 77 deletions(-)
> rename arch/powerpc/mm/book3s64/{hash_hugetlbpage.c => hugetlbpage.c} (99%)
>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index 497b7d0b2d7e..50c361b8b7fd 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -963,6 +963,7 @@ config PPC_MEM_KEYS
> prompt "PowerPC Memory Protection Keys"
> def_bool y
> depends on PPC_BOOK3S_64
> + depends on PPC_HASH_MMU
> select ARCH_USES_HIGH_VMA_FLAGS
> select ARCH_HAS_PKEYS
> help
> diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
> index bb3deb76c951..fca69dd23e25 100644
> --- a/arch/powerpc/include/asm/book3s/64/mmu.h
> +++ b/arch/powerpc/include/asm/book3s/64/mmu.h
> @@ -107,7 +107,9 @@ typedef struct {
> * from EA and new context ids to build the new VAs.
> */
> mm_context_id_t id;
> +#ifdef CONFIG_PPC_HASH_MMU
> mm_context_id_t extended_id[TASK_SIZE_USER64/TASK_CONTEXT_SIZE];
> +#endif
> };
>
> /* Number of bits in the mm_cpumask */
> @@ -116,7 +118,9 @@ typedef struct {
> /* Number of users of the external (Nest) MMU */
> atomic_t copros;
>
> +#ifdef CONFIG_PPC_HASH_MMU
> struct hash_mm_context *hash_context;
> +#endif
>
> unsigned long vdso_base;
> /*
> @@ -139,6 +143,7 @@ typedef struct {
> #endif
> } mm_context_t;
>
> +#ifdef CONFIG_PPC_HASH_MMU
> static inline u16 mm_ctx_user_psize(mm_context_t *ctx)
> {
> return ctx->hash_context->user_psize;
> @@ -193,11 +198,22 @@ static inline struct subpage_prot_table *mm_ctx_subpage_prot(mm_context_t *ctx)
> }
> #endif
>
> +#endif
> +
> /*
> * The current system page and segment sizes
> */
> -extern int mmu_linear_psize;
> +#if defined(CONFIG_PPC_RADIX_MMU) && !defined(CONFIG_PPC_HASH_MMU)
> +#ifdef CONFIG_PPC_64K_PAGES
> +#define mmu_virtual_psize MMU_PAGE_64K
> +#else
> +#define mmu_virtual_psize MMU_PAGE_4K
> +#endif
> +#else
> extern int mmu_virtual_psize;
> +#endif
> +
> +extern int mmu_linear_psize;
> extern int mmu_vmalloc_psize;
> extern int mmu_vmemmap_psize;
> extern int mmu_io_psize;
> @@ -243,6 +259,7 @@ extern void radix_init_pseries(void);
> static inline void radix_init_pseries(void) { };
> #endif
>
> +#ifdef CONFIG_PPC_HASH_MMU
> static inline int get_user_context(mm_context_t *ctx, unsigned long ea)
> {
> int index = ea >> MAX_EA_BITS_PER_CONTEXT;
> @@ -262,6 +279,7 @@ static inline unsigned long get_user_vsid(mm_context_t *ctx,
>
> return get_vsid(context, ea, ssize);
> }
> +#endif
>
> #endif /* __ASSEMBLY__ */
> #endif /* _ASM_POWERPC_BOOK3S_64_MMU_H_ */
> diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
> index 201a69e6a355..0d0b5dcadd16 100644
> --- a/arch/powerpc/include/asm/book3s/64/pgtable.h
> +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
> @@ -1139,8 +1139,14 @@ extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot);
> extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot);
> extern void set_pmd_at(struct mm_struct *mm, unsigned long addr,
> pmd_t *pmdp, pmd_t pmd);
> +#ifdef CONFIG_PPC_HASH_MMU
> extern void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr,
> pmd_t *pmd);
> +#else
> +static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd)
> +{
> +}
> +#endif
> extern int hash__has_transparent_hugepage(void);
> static inline int has_transparent_hugepage(void)
> {
> diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
> index 64d02a704bcb..bb8a22ee9668 100644
> --- a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
> +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
> @@ -112,9 +112,15 @@ static inline void hash__flush_tlb_kernel_range(unsigned long start,
>
> struct mmu_gather;
> extern void hash__tlb_flush(struct mmu_gather *tlb);
> +extern void flush_tlb_pmd_range(struct mm_struct *mm, pmd_t *pmd,
> + unsigned long addr);
> +
> +#ifdef CONFIG_PPC_HASH_MMU
> /* Private function for use by PCI IO mapping code */
> extern void __flush_hash_table_range(struct mm_struct *mm, unsigned long start,
> unsigned long end);
> -extern void flush_tlb_pmd_range(struct mm_struct *mm, pmd_t *pmd,
> - unsigned long addr);
> +#else
> +static inline void __flush_hash_table_range(struct mm_struct *mm, unsigned long start,
> + unsigned long end) { }
> +#endif
> #endif /* _ASM_POWERPC_BOOK3S_64_TLBFLUSH_HASH_H */
> diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h
> index dcb5c3839d2f..d2e80f178b6d 100644
> --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h
> +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h
> @@ -14,7 +14,6 @@ enum {
> TLB_INVAL_SCOPE_LPID = 1, /* invalidate TLBs for current LPID */
> };
>
> -#ifdef CONFIG_PPC_NATIVE
> static inline void tlbiel_all(void)
> {
> /*
> @@ -30,9 +29,6 @@ static inline void tlbiel_all(void)
> else
> hash__tlbiel_all(TLB_INVAL_SCOPE_GLOBAL);
> }
> -#else
> -static inline void tlbiel_all(void) { BUG(); };
> -#endif
>
> static inline void tlbiel_all_lpid(bool radix)
> {
> diff --git a/arch/powerpc/include/asm/book3s/pgtable.h b/arch/powerpc/include/asm/book3s/pgtable.h
> index 0e1263455d73..cb85da522d0c 100644
> --- a/arch/powerpc/include/asm/book3s/pgtable.h
> +++ b/arch/powerpc/include/asm/book3s/pgtable.h
> @@ -26,6 +26,7 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
> unsigned long size, pgprot_t vma_prot);
> #define __HAVE_PHYS_MEM_ACCESS_PROT
>
> +#ifdef CONFIG_PPC_HASH_MMU
> /*
> * This gets called at the end of handling a page fault, when
> * the kernel has put a new PTE into the page table for the process.
> @@ -36,6 +37,9 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
> * waiting for the inevitable extra hash-table miss exception.
> */
> void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep);
> +#else
> +static inline void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) {}
> +#endif
>
> #endif /* __ASSEMBLY__ */
> #endif
> diff --git a/arch/powerpc/include/asm/mmu.h b/arch/powerpc/include/asm/mmu.h
> index 0699cfeeb8c9..9ff4b99cd0d3 100644
> --- a/arch/powerpc/include/asm/mmu.h
> +++ b/arch/powerpc/include/asm/mmu.h
> @@ -269,7 +269,7 @@ static inline void assert_pte_locked(struct mm_struct *mm, unsigned long addr)
> }
> #endif /* !CONFIG_DEBUG_VM */
>
> -#ifdef CONFIG_PPC_RADIX_MMU
> +#if defined(CONFIG_PPC_RADIX_MMU) && defined(CONFIG_PPC_HASH_MMU)
> static inline bool radix_enabled(void)
> {
> return mmu_has_feature(MMU_FTR_TYPE_RADIX);
> @@ -279,7 +279,7 @@ static inline bool early_radix_enabled(void)
> {
> return early_mmu_has_feature(MMU_FTR_TYPE_RADIX);
> }
> -#else
> +#elif defined(CONFIG_PPC_HASH_MMU)
> static inline bool radix_enabled(void)
> {
> return false;
> @@ -289,6 +289,16 @@ static inline bool early_radix_enabled(void)
> {
> return false;
> }
> +#else
> +static inline bool radix_enabled(void)
> +{
> + return true;
> +}
> +
> +static inline bool early_radix_enabled(void)
> +{
> + return true;
> +}
> #endif
>
> #ifdef CONFIG_PPC_MEM_KEYS
> diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h
> index 360367c579de..6a0fafbef119 100644
> --- a/arch/powerpc/include/asm/mmu_context.h
> +++ b/arch/powerpc/include/asm/mmu_context.h
> @@ -74,6 +74,7 @@ extern void hash__reserve_context_id(int id);
> extern void __destroy_context(int context_id);
> static inline void mmu_context_init(void) { }
>
> +#ifdef CONFIG_PPC_HASH_MMU
> static inline int alloc_extended_context(struct mm_struct *mm,
> unsigned long ea)
> {
> @@ -99,6 +100,7 @@ static inline bool need_extra_context(struct mm_struct *mm, unsigned long ea)
> return true;
> return false;
> }
> +#endif
>
> #else
> extern void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next,
> diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
> index e3cc9eb9204d..6b365cffaee9 100644
> --- a/arch/powerpc/include/asm/paca.h
> +++ b/arch/powerpc/include/asm/paca.h
> @@ -95,7 +95,9 @@ struct paca_struct {
> /* this becomes non-zero. */
> u8 kexec_state; /* set when kexec down has irqs off */
> #ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> struct slb_shadow *slb_shadow_ptr;
> +#endif
> struct dtl_entry *dispatch_log;
> struct dtl_entry *dispatch_log_end;
> #endif
> @@ -109,6 +111,8 @@ struct paca_struct {
> u64 exgen[EX_SIZE] __attribute__((aligned(0x80)));
> u64 exslb[EX_SIZE]; /* used for SLB/segment table misses
> * on the linear mapping */
> +
> +#ifdef CONFIG_PPC_HASH_MMU
> /* SLB related definitions */
> u16 vmalloc_sllp;
> u8 slb_cache_ptr;
> @@ -119,6 +123,7 @@ struct paca_struct {
> u32 slb_used_bitmap; /* Bitmaps for first 32 SLB entries. */
> u32 slb_kern_bitmap;
> u32 slb_cache[SLB_CACHE_ENTRIES];
> +#endif
> #endif /* CONFIG_PPC_BOOK3S_64 */
>
> #ifdef CONFIG_PPC_BOOK3E
> @@ -149,6 +154,7 @@ struct paca_struct {
>
> #ifdef CONFIG_PPC_BOOK3S
> mm_context_id_t mm_ctx_id;
> +#ifdef CONFIG_PPC_HASH_MMU
> #ifdef CONFIG_PPC_MM_SLICES
> unsigned char mm_ctx_low_slices_psize[BITS_PER_LONG / BITS_PER_BYTE];
> unsigned char mm_ctx_high_slices_psize[SLICE_ARRAY_SIZE];
> @@ -157,6 +163,7 @@ struct paca_struct {
> u16 mm_ctx_user_psize;
> u16 mm_ctx_sllp;
> #endif
> +#endif
> #endif
>
> /*
> @@ -260,9 +267,11 @@ struct paca_struct {
> #endif /* CONFIG_PPC_PSERIES */
>
> #ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> /* Capture SLB related old contents in MCE handler. */
> struct slb_entry *mce_faulty_slbs;
> u16 slb_save_cache_ptr;
> +#endif
> #endif /* CONFIG_PPC_BOOK3S_64 */
> #ifdef CONFIG_STACKPROTECTOR
> unsigned long canary;
> diff --git a/arch/powerpc/include/asm/sparsemem.h b/arch/powerpc/include/asm/sparsemem.h
> index 3192d454a733..958fed1f052e 100644
> --- a/arch/powerpc/include/asm/sparsemem.h
> +++ b/arch/powerpc/include/asm/sparsemem.h
> @@ -16,7 +16,7 @@
> extern int create_section_mapping(unsigned long start, unsigned long end, int nid);
> extern int remove_section_mapping(unsigned long start, unsigned long end);
>
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> extern int resize_hpt_for_hotplug(unsigned long new_mem_size);
> #else
> static inline int resize_hpt_for_hotplug(unsigned long new_mem_size) { return 0; }
> diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
> index fcf24a365fc0..c4a8a070eab0 100644
> --- a/arch/powerpc/kernel/asm-offsets.c
> +++ b/arch/powerpc/kernel/asm-offsets.c
> @@ -238,6 +238,7 @@ int main(void)
> #endif /* CONFIG_PPC_BOOK3E */
>
> #ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> OFFSET(PACASLBCACHE, paca_struct, slb_cache);
> OFFSET(PACASLBCACHEPTR, paca_struct, slb_cache_ptr);
> OFFSET(PACASTABRR, paca_struct, stab_rr);
> @@ -246,6 +247,7 @@ int main(void)
> OFFSET(MMUPSIZESLLP, mmu_psize_def, sllp);
> #else
> OFFSET(PACACONTEXTSLLP, paca_struct, mm_ctx_sllp);
> +#endif
> #endif /* CONFIG_PPC_MM_SLICES */
> OFFSET(PACA_EXGEN, paca_struct, exgen);
> OFFSET(PACA_EXMC, paca_struct, exmc);
> @@ -254,10 +256,12 @@ int main(void)
> #ifdef CONFIG_PPC_PSERIES
> OFFSET(PACALPPACAPTR, paca_struct, lppaca_ptr);
> #endif
> +#ifdef CONFIG_PPC_HASH_MMU
> OFFSET(PACA_SLBSHADOWPTR, paca_struct, slb_shadow_ptr);
> OFFSET(SLBSHADOW_STACKVSID, slb_shadow, save_area[SLB_NUM_BOLTED - 1].vsid);
> OFFSET(SLBSHADOW_STACKESID, slb_shadow, save_area[SLB_NUM_BOLTED - 1].esid);
> OFFSET(SLBSHADOW_SAVEAREA, slb_shadow, save_area);
> +#endif
> OFFSET(LPPACA_PMCINUSE, lppaca, pmcregs_in_use);
> #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
> OFFSET(PACA_PMCINUSE, paca_struct, pmcregs_in_use);
> diff --git a/arch/powerpc/kernel/dt_cpu_ftrs.c b/arch/powerpc/kernel/dt_cpu_ftrs.c
> index 182b4047c1ef..36385eb87d36 100644
> --- a/arch/powerpc/kernel/dt_cpu_ftrs.c
> +++ b/arch/powerpc/kernel/dt_cpu_ftrs.c
> @@ -298,6 +298,7 @@ static int __init feat_enable_idle_stop(struct dt_cpu_feature *f)
>
> static int __init feat_enable_mmu_hash(struct dt_cpu_feature *f)
> {
> +#ifdef CONFIG_PPC_HASH_MMU
> u64 lpcr;
>
> lpcr = mfspr(SPRN_LPCR);
> @@ -313,10 +314,13 @@ static int __init feat_enable_mmu_hash(struct dt_cpu_feature *f)
> cur_cpu_spec->cpu_user_features |= PPC_FEATURE_HAS_MMU;
>
> return 1;
> +#endif
> + return 0;
> }
>
> static int __init feat_enable_mmu_hash_v3(struct dt_cpu_feature *f)
> {
> +#ifdef CONFIG_PPC_HASH_MMU
> u64 lpcr;
>
> system_registers.lpcr_clear |= (LPCR_ISL | LPCR_UPRT | LPCR_HR);
> @@ -328,14 +332,18 @@ static int __init feat_enable_mmu_hash_v3(struct dt_cpu_feature *f)
> cur_cpu_spec->cpu_user_features |= PPC_FEATURE_HAS_MMU;
>
> return 1;
> +#endif
> + return 0;
> }
>
>
> static int __init feat_enable_mmu_radix(struct dt_cpu_feature *f)
> {
> +#ifdef CONFIG_PPC_HASH_MMU
> + cur_cpu_spec->mmu_features |= MMU_FTRS_HASH_BASE;
> +#endif
> #ifdef CONFIG_PPC_RADIX_MMU
> cur_cpu_spec->mmu_features |= MMU_FTR_TYPE_RADIX;
> - cur_cpu_spec->mmu_features |= MMU_FTRS_HASH_BASE;
> cur_cpu_spec->cpu_user_features |= PPC_FEATURE_HAS_MMU;
>
> return 1;
> diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
> index 6ba675b0cf7d..5f40fe4ec6ba 100644
> --- a/arch/powerpc/kernel/entry_64.S
> +++ b/arch/powerpc/kernel/entry_64.S
> @@ -646,7 +646,7 @@ _GLOBAL(_switch)
> #endif
>
> ld r8,KSP(r4) /* new stack pointer */
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> BEGIN_MMU_FTR_SECTION
> b 2f
> END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX)
> @@ -698,7 +698,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S)
> slbmte r7,r0
> isync
> 2:
> -#endif /* CONFIG_PPC_BOOK3S_64 */
> +#endif /* CONFIG_PPC_HASH_MMU */
>
> clrrdi r7, r8, THREAD_SHIFT /* base of new stack */
> /* Note: this uses SWITCH_FRAME_SIZE rather than INT_FRAME_SIZE
> diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
> index ffc15f4f079d..b94874c2f7ab 100644
> --- a/arch/powerpc/kernel/exceptions-64s.S
> +++ b/arch/powerpc/kernel/exceptions-64s.S
> @@ -1161,6 +1161,7 @@ EXC_COMMON_BEGIN(data_access_common)
> INT_COMMON 0x300, PACA_EXGEN, 1, 1, 1, 1, 1
> ld r4,_DAR(r1)
> ld r5,_DSISR(r1)
> +#ifdef CONFIG_PPC_HASH_MMU
> BEGIN_MMU_FTR_SECTION
> ld r6,_MSR(r1)
> li r3,0x300
> @@ -1168,6 +1169,9 @@ BEGIN_MMU_FTR_SECTION
> MMU_FTR_SECTION_ELSE
> b handle_page_fault
> ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
> +#else
> + b handle_page_fault
> +#endif
>
>
> EXC_REAL_BEGIN(data_access_slb, 0x380, 0x80)
> @@ -1181,6 +1185,7 @@ EXC_COMMON_BEGIN(data_access_slb_common)
> INT_COMMON 0x380, PACA_EXSLB, 1, 1, 0, 1, 0
> ld r4,_DAR(r1)
> addi r3,r1,STACK_FRAME_OVERHEAD
> +#ifdef CONFIG_PPC_HASH_MMU
> BEGIN_MMU_FTR_SECTION
> /* HPT case, do SLB fault */
> bl do_slb_fault
> @@ -1192,6 +1197,9 @@ MMU_FTR_SECTION_ELSE
> /* Radix case, access is outside page table range */
> li r3,-EFAULT
> ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
> +#else
> + li r3,-EFAULT
> +#endif
> std r3,RESULT(r1)
> bl save_nvgprs
> RECONCILE_IRQ_STATE(r10, r11)
> @@ -1213,6 +1221,7 @@ EXC_COMMON_BEGIN(instruction_access_common)
> INT_COMMON 0x400, PACA_EXGEN, 1, 1, 1, 2, 2
> ld r4,_DAR(r1)
> ld r5,_DSISR(r1)
> +#ifdef CONFIG_PPC_HASH_MMU
> BEGIN_MMU_FTR_SECTION
> ld r6,_MSR(r1)
> li r3,0x400
> @@ -1220,6 +1229,9 @@ BEGIN_MMU_FTR_SECTION
> MMU_FTR_SECTION_ELSE
> b handle_page_fault
> ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
> +#else
> + b handle_page_fault
> +#endif
>
>
> EXC_REAL_BEGIN(instruction_access_slb, 0x480, 0x80)
> @@ -1233,6 +1245,7 @@ EXC_COMMON_BEGIN(instruction_access_slb_common)
> INT_COMMON 0x480, PACA_EXSLB, 1, 1, 0, 2, 0
> ld r4,_DAR(r1)
> addi r3,r1,STACK_FRAME_OVERHEAD
> +#ifdef CONFIG_PPC_HASH_MMU
> BEGIN_MMU_FTR_SECTION
> /* HPT case, do SLB fault */
> bl do_slb_fault
> @@ -1244,6 +1257,9 @@ MMU_FTR_SECTION_ELSE
> /* Radix case, access is outside page table range */
> li r3,-EFAULT
> ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
> +#else
> + li r3,-EFAULT
> +#endif
> std r3,RESULT(r1)
> bl save_nvgprs
> RECONCILE_IRQ_STATE(r10, r11)
> @@ -2252,8 +2268,8 @@ disable_machine_check:
> * Hash table stuff
> */
> .balign IFETCH_ALIGN_BYTES
> +#ifdef CONFIG_PPC_HASH_MMU
> do_hash_page:
> -#ifdef CONFIG_PPC_BOOK3S_64
> lis r0,(DSISR_BAD_FAULT_64S | DSISR_DABRMATCH | DSISR_KEYFAULT)@h
> ori r0,r0,DSISR_BAD_FAULT_64S@l
> and. r0,r5,r0 /* weird error? */
> @@ -2283,7 +2299,7 @@ do_hash_page:
> /* Reload DAR/DSISR into r4/r5 for the DABR check below */
> ld r4,_DAR(r1)
> ld r5,_DSISR(r1)
> -#endif /* CONFIG_PPC_BOOK3S_64 */
> +#endif /* CONFIG_PPC_HASH_MMU */
>
> /* Here we have a page fault that hash_page can't handle. */
> handle_page_fault:
> @@ -2315,7 +2331,7 @@ handle_dabr_fault:
> b ret_from_except
>
>
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> /* We have a page fault that hash_page could handle but HV refused
> * the PTE insertion
> */
> diff --git a/arch/powerpc/kernel/mce.c b/arch/powerpc/kernel/mce.c
> index 34c1001e9e8b..c8ff446acec0 100644
> --- a/arch/powerpc/kernel/mce.c
> +++ b/arch/powerpc/kernel/mce.c
> @@ -544,7 +544,7 @@ void machine_check_print_event_info(struct machine_check_event *evt,
> mc_error_class[evt->error_class] : "Unknown";
> printk("%sMCE: CPU%d: %s\n", level, evt->cpu, subtype);
>
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> /* Display faulty slb contents for SLB errors. */
> if (evt->error_type == MCE_ERROR_TYPE_SLB)
> slb_dump_contents(local_paca->mce_faulty_slbs);
> diff --git a/arch/powerpc/kernel/mce_power.c b/arch/powerpc/kernel/mce_power.c
> index 1cbf7f1a4e3d..fd03ac9d7f7b 100644
> --- a/arch/powerpc/kernel/mce_power.c
> +++ b/arch/powerpc/kernel/mce_power.c
> @@ -58,7 +58,7 @@ unsigned long addr_to_pfn(struct pt_regs *regs, unsigned long addr)
> }
>
> /* flush SLBs and reload */
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> void flush_and_reload_slb(void)
> {
> /* Invalidate all SLBs */
> @@ -88,7 +88,7 @@ void flush_and_reload_slb(void)
>
> static void flush_erat(void)
> {
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> if (!early_cpu_has_feature(CPU_FTR_ARCH_300)) {
> flush_and_reload_slb();
> return;
> @@ -103,7 +103,7 @@ static void flush_erat(void)
>
> static int mce_flush(int what)
> {
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> if (what == MCE_FLUSH_SLB) {
> flush_and_reload_slb();
> return 1;
> @@ -409,8 +409,10 @@ static int mce_handle_ierror(struct pt_regs *regs,
> /* attempt to correct the error */
> switch (table[i].error_type) {
> case MCE_ERROR_TYPE_SLB:
> +#ifdef CONFIG_PPC_HASH_MMU
> if (local_paca->in_mce == 1)
> slb_save_contents(local_paca->mce_faulty_slbs);
> +#endif
> handled = mce_flush(MCE_FLUSH_SLB);
> break;
> case MCE_ERROR_TYPE_ERAT:
> @@ -496,8 +498,10 @@ static int mce_handle_derror(struct pt_regs *regs,
> /* attempt to correct the error */
> switch (table[i].error_type) {
> case MCE_ERROR_TYPE_SLB:
> +#ifdef CONFIG_PPC_HASH_MMU
> if (local_paca->in_mce == 1)
> slb_save_contents(local_paca->mce_faulty_slbs);
> +#endif
> if (mce_flush(MCE_FLUSH_SLB))
> handled = 1;
> break;
> diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
> index 949eceb254d8..415bdd687900 100644
> --- a/arch/powerpc/kernel/paca.c
> +++ b/arch/powerpc/kernel/paca.c
> @@ -131,8 +131,7 @@ static struct lppaca * __init new_lppaca(int cpu, unsigned long limit)
> }
> #endif /* CONFIG_PPC_PSERIES */
>
> -#ifdef CONFIG_PPC_BOOK3S_64
> -
> +#ifdef CONFIG_PPC_HASH_MMU
> /*
> * 3 persistent SLBs are allocated here. The buffer will be zero
> * initially, hence will all be invaild until we actually write them.
> @@ -161,8 +160,7 @@ static struct slb_shadow * __init new_slb_shadow(int cpu, unsigned long limit)
>
> return s;
> }
> -
> -#endif /* CONFIG_PPC_BOOK3S_64 */
> +#endif /* CONFIG_PPC_HASH_MMU */
>
> /* The Paca is an array with one entry per processor. Each contains an
> * lppaca, which contains the information shared between the
> @@ -194,7 +192,7 @@ void __init initialise_paca(struct paca_struct *new_paca, int cpu)
> new_paca->kexec_state = KEXEC_STATE_NONE;
> new_paca->__current = &init_task;
> new_paca->data_offset = 0xfeeeeeeeeeeeeeeeULL;
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> new_paca->slb_shadow_ptr = NULL;
> #endif
>
> @@ -267,7 +265,7 @@ void __init allocate_paca(int cpu)
> #ifdef CONFIG_PPC_PSERIES
> paca->lppaca_ptr = new_lppaca(cpu, limit);
> #endif
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> paca->slb_shadow_ptr = new_slb_shadow(cpu, limit);
> #endif
> paca_struct_size += sizeof(struct paca_struct);
> @@ -285,7 +283,7 @@ void __init free_unused_pacas(void)
> paca_nr_cpu_ids = nr_cpu_ids;
> paca_ptrs_size = new_ptrs_size;
>
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> if (early_radix_enabled()) {
> /* Ugly fixup, see new_slb_shadow() */
> memblock_free(__pa(paca_ptrs[boot_cpuid]->slb_shadow_ptr),
> @@ -298,9 +296,9 @@ void __init free_unused_pacas(void)
> paca_ptrs_size + paca_struct_size, nr_cpu_ids);
> }
>
> +#ifdef CONFIG_PPC_HASH_MMU
> void copy_mm_to_paca(struct mm_struct *mm)
> {
> -#ifdef CONFIG_PPC_BOOK3S
> mm_context_t *context = &mm->context;
>
> get_paca()->mm_ctx_id = context->id;
> @@ -315,7 +313,5 @@ void copy_mm_to_paca(struct mm_struct *mm)
> get_paca()->mm_ctx_user_psize = context->user_psize;
> get_paca()->mm_ctx_sllp = context->sllp;
> #endif
> -#else /* !CONFIG_PPC_BOOK3S */
> - return;
> -#endif
> }
> +#endif /* CONFIG_PPC_HASH_MMU */
> diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
> index fad50db9dcf2..c2e8eb100a89 100644
> --- a/arch/powerpc/kernel/process.c
> +++ b/arch/powerpc/kernel/process.c
> @@ -1140,7 +1140,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
> {
> struct thread_struct *new_thread, *old_thread;
> struct task_struct *last;
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#if defined(CONFIG_PPC_BOOK3S_64) && defined(CONFIG_PPC_HASH_MMU)
> struct ppc64_tlb_batch *batch;
> #endif
>
> @@ -1149,7 +1149,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
>
> WARN_ON(!irqs_disabled());
>
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#if defined(CONFIG_PPC_BOOK3S_64) && defined(CONFIG_PPC_HASH_MMU)
> batch = this_cpu_ptr(&ppc64_tlb_batch);
> if (batch->active) {
> current_thread_info()->local_flags |= _TLF_LAZY_MMU;
> @@ -1204,11 +1204,13 @@ struct task_struct *__switch_to(struct task_struct *prev,
> last = _switch(old_thread, new_thread);
>
> #ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> if (current_thread_info()->local_flags & _TLF_LAZY_MMU) {
> current_thread_info()->local_flags &= ~_TLF_LAZY_MMU;
> batch = this_cpu_ptr(&ppc64_tlb_batch);
> batch->active = 1;
> }
> +#endif
>
> if (current->thread.regs) {
> restore_math(current->thread.regs);
> @@ -1568,7 +1570,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
>
> static void setup_ksp_vsid(struct task_struct *p, unsigned long sp)
> {
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#if defined(CONFIG_PPC_BOOK3S_64) && defined(CONFIG_PPC_HASH_MMU)
> unsigned long sp_vsid;
> unsigned long llp = mmu_psize_defs[mmu_linear_psize].sllp;
>
> @@ -2179,10 +2181,9 @@ unsigned long arch_randomize_brk(struct mm_struct *mm)
> * the heap, we can put it above 1TB so it is backed by a 1TB
> * segment. Otherwise the heap will be in the bottom 1TB
> * which always uses 256MB segments and this may result in a
> - * performance penalty. We don't need to worry about radix. For
> - * radix, mmu_highuser_ssize remains unchanged from 256MB.
> + * performance penalty.
> */
> - if (!is_32bit_task() && (mmu_highuser_ssize == MMU_SEGSIZE_1T))
> + if (!radix_enabled() && !is_32bit_task() && (mmu_highuser_ssize == MMU_SEGSIZE_1T))
> base = max_t(unsigned long, mm->brk, 1UL << SID_SHIFT_1T);
> #endif
>
> diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c
> index 6620f37abe73..f372340cbcd6 100644
> --- a/arch/powerpc/kernel/prom.c
> +++ b/arch/powerpc/kernel/prom.c
> @@ -232,6 +232,7 @@ static void __init check_cpu_pa_features(unsigned long node)
> #ifdef CONFIG_PPC_BOOK3S_64
> static void __init init_mmu_slb_size(unsigned long node)
> {
> +#ifdef CONFIG_PPC_HASH_MMU
> const __be32 *slb_size_ptr;
>
> slb_size_ptr = of_get_flat_dt_prop(node, "slb-size", NULL) ? :
> @@ -239,6 +240,7 @@ static void __init init_mmu_slb_size(unsigned long node)
>
> if (slb_size_ptr)
> mmu_slb_size = be32_to_cpup(slb_size_ptr);
> +#endif
> }
> #else
> #define init_mmu_slb_size(node) do { } while(0)
> diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
> index e05e6dd67ae6..70f2db84f5f4 100644
> --- a/arch/powerpc/kernel/setup_64.c
> +++ b/arch/powerpc/kernel/setup_64.c
> @@ -758,6 +758,7 @@ void __init setup_per_cpu_areas(void)
> unsigned int cpu;
> int rc;
>
> +#ifdef CONFIG_PPC_HASH_MMU
> /*
> * Linear mapping is one of 4K, 1M and 16M. For 4K, no need
> * to group units. For larger mappings, use 1M atom which
> @@ -767,6 +768,9 @@ void __init setup_per_cpu_areas(void)
> atom_size = PAGE_SIZE;
> else
> atom_size = 1 << 20;
> +#else
> + atom_size = PAGE_SIZE;
> +#endif
>
> rc = pcpu_embed_first_chunk(0, dyn_size, atom_size, pcpu_cpu_distance,
> pcpu_fc_alloc, pcpu_fc_free);
> diff --git a/arch/powerpc/kexec/core_64.c b/arch/powerpc/kexec/core_64.c
> index 04a7cba58eff..97b580b0a766 100644
> --- a/arch/powerpc/kexec/core_64.c
> +++ b/arch/powerpc/kexec/core_64.c
> @@ -372,7 +372,7 @@ void default_machine_kexec(struct kimage *image)
> /* NOTREACHED */
> }
>
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> /* Values we need to export to the second kernel via the device tree. */
> static unsigned long htab_base;
> static unsigned long htab_size;
> @@ -414,4 +414,4 @@ static int __init export_htab_values(void)
> return 0;
> }
> late_initcall(export_htab_values);
> -#endif /* CONFIG_PPC_BOOK3S_64 */
> +#endif /* CONFIG_PPC_HASH_MMU */
> diff --git a/arch/powerpc/kvm/Kconfig b/arch/powerpc/kvm/Kconfig
> index 711fca9bc6f0..1693f1eebea8 100644
> --- a/arch/powerpc/kvm/Kconfig
> +++ b/arch/powerpc/kvm/Kconfig
> @@ -70,6 +70,7 @@ config KVM_BOOK3S_64
> select KVM
> select KVM_BOOK3S_PR_POSSIBLE if !KVM_BOOK3S_HV_POSSIBLE
> select SPAPR_TCE_IOMMU if IOMMU_SUPPORT && (PPC_PSERIES || PPC_POWERNV)
> + select PPC_HASH_MMU
> ---help---
> Support running unmodified book3s_64 and book3s_32 guest kernels
> in virtual machines on book3s_64 host processors.
> diff --git a/arch/powerpc/mm/book3s64/Makefile b/arch/powerpc/mm/book3s64/Makefile
> index fd393b8be14f..5376e11ec85f 100644
> --- a/arch/powerpc/mm/book3s64/Makefile
> +++ b/arch/powerpc/mm/book3s64/Makefile
> @@ -2,20 +2,23 @@
>
> ccflags-y := $(NO_MINIMAL_TOC)
>
> +obj-y += mmu_context.o pgtable.o
> +ifdef CONFIG_PPC_HASH_MMU
> CFLAGS_REMOVE_slb.o = $(CC_FLAGS_FTRACE)
> -
> -obj-y += hash_pgtable.o hash_utils.o slb.o \
> - mmu_context.o pgtable.o hash_tlb.o
> +obj-y += hash_pgtable.o hash_utils.o hash_tlb.o slb.o
> obj-$(CONFIG_PPC_NATIVE) += hash_native.o
> -obj-$(CONFIG_PPC_RADIX_MMU) += radix_pgtable.o radix_tlb.o
> obj-$(CONFIG_PPC_4K_PAGES) += hash_4k.o
> obj-$(CONFIG_PPC_64K_PAGES) += hash_64k.o
> -obj-$(CONFIG_HUGETLB_PAGE) += hash_hugetlbpage.o
> +obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += hash_hugepage.o
> +obj-$(CONFIG_PPC_SUBPAGE_PROT) += subpage_prot.o
> +endif
> +
> +obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
> +
> +obj-$(CONFIG_PPC_RADIX_MMU) += radix_pgtable.o radix_tlb.o
> ifdef CONFIG_HUGETLB_PAGE
> obj-$(CONFIG_PPC_RADIX_MMU) += radix_hugetlbpage.o
> endif
> -obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += hash_hugepage.o
> -obj-$(CONFIG_PPC_SUBPAGE_PROT) += subpage_prot.o
> obj-$(CONFIG_SPAPR_TCE_IOMMU) += iommu_api.o
> obj-$(CONFIG_PPC_MEM_KEYS) += pkeys.o
>
> diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c
> index 64733b9cb20a..637f90452657 100644
> --- a/arch/powerpc/mm/book3s64/hash_pgtable.c
> +++ b/arch/powerpc/mm/book3s64/hash_pgtable.c
> @@ -17,7 +17,6 @@
>
> #include <mm/mmu_decl.h>
>
> -#define CREATE_TRACE_POINTS
> #include <trace/events/thp.h>
>
> #if H_PGTABLE_RANGE > (USER_VSID_RANGE * (TASK_SIZE_USER64 / TASK_CONTEXT_SIZE))
> diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c
> index 523d4d39d11e..44ade204d050 100644
> --- a/arch/powerpc/mm/book3s64/hash_utils.c
> +++ b/arch/powerpc/mm/book3s64/hash_utils.c
> @@ -95,8 +95,6 @@
> */
>
> static unsigned long _SDR1;
> -struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT];
> -EXPORT_SYMBOL_GPL(mmu_psize_defs);
>
> u8 hpte_page_sizes[1 << LP_BITS];
> EXPORT_SYMBOL_GPL(hpte_page_sizes);
> @@ -105,14 +103,7 @@ struct hash_pte *htab_address;
> unsigned long htab_size_bytes;
> unsigned long htab_hash_mask;
> EXPORT_SYMBOL_GPL(htab_hash_mask);
> -int mmu_linear_psize = MMU_PAGE_4K;
> -EXPORT_SYMBOL_GPL(mmu_linear_psize);
> int mmu_virtual_psize = MMU_PAGE_4K;
> -int mmu_vmalloc_psize = MMU_PAGE_4K;
> -#ifdef CONFIG_SPARSEMEM_VMEMMAP
> -int mmu_vmemmap_psize = MMU_PAGE_4K;
> -#endif
> -int mmu_io_psize = MMU_PAGE_4K;
> int mmu_kernel_ssize = MMU_SEGSIZE_256M;
> EXPORT_SYMBOL_GPL(mmu_kernel_ssize);
> int mmu_highuser_ssize = MMU_SEGSIZE_256M;
> diff --git a/arch/powerpc/mm/book3s64/hash_hugetlbpage.c b/arch/powerpc/mm/book3s64/hugetlbpage.c
> similarity index 99%
> rename from arch/powerpc/mm/book3s64/hash_hugetlbpage.c
> rename to arch/powerpc/mm/book3s64/hugetlbpage.c
> index eefa89c6117b..e4df7016e0fb 100644
> --- a/arch/powerpc/mm/book3s64/hash_hugetlbpage.c
> +++ b/arch/powerpc/mm/book3s64/hugetlbpage.c
> @@ -18,6 +18,7 @@
> unsigned int hpage_shift;
> EXPORT_SYMBOL(hpage_shift);
>
> +#ifdef CONFIG_PPC_HASH_MMU
> extern long hpte_insert_repeating(unsigned long hash, unsigned long vpn,
> unsigned long pa, unsigned long rlags,
> unsigned long vflags, int psize, int ssize);
> @@ -128,6 +129,7 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,
> *ptep = __pte(new_pte & ~H_PAGE_BUSY);
> return 0;
> }
> +#endif
>
> pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma,
> unsigned long addr, pte_t *ptep)
> diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c
> index 0ba30b8b935b..660941edd550 100644
> --- a/arch/powerpc/mm/book3s64/mmu_context.c
> +++ b/arch/powerpc/mm/book3s64/mmu_context.c
> @@ -28,6 +28,7 @@ static int alloc_context_id(int min_id, int max_id)
> return ida_alloc_range(&mmu_context_ida, min_id, max_id, GFP_KERNEL);
> }
>
> +#ifdef CONFIG_PPC_HASH_MMU
> void hash__reserve_context_id(int id)
> {
> int result = ida_alloc_range(&mmu_context_ida, id, id, GFP_KERNEL);
> @@ -47,9 +48,9 @@ int hash__alloc_context_id(void)
> return alloc_context_id(MIN_USER_CONTEXT, max);
> }
> EXPORT_SYMBOL_GPL(hash__alloc_context_id);
> +#endif
>
> -void slb_setup_new_exec(void);
> -
> +#ifdef CONFIG_PPC_HASH_MMU
> static int realloc_context_ids(mm_context_t *ctx)
> {
> int i, id;
> @@ -143,12 +144,15 @@ static int hash__init_new_context(struct mm_struct *mm)
> return index;
> }
>
> +void slb_setup_new_exec(void);
> +
> void hash__setup_new_exec(void)
> {
> slice_setup_new_exec();
>
> slb_setup_new_exec();
> }
> +#endif
>
> static int radix__init_new_context(struct mm_struct *mm)
> {
> @@ -174,7 +178,9 @@ static int radix__init_new_context(struct mm_struct *mm)
> */
> asm volatile("ptesync;isync" : : : "memory");
>
> +#ifdef CONFIG_PPC_HASH_MMU
> mm->context.hash_context = NULL;
> +#endif
>
> return index;
> }
> @@ -185,8 +191,10 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
>
> if (radix_enabled())
> index = radix__init_new_context(mm);
> +#ifdef CONFIG_PPC_HASH_MMU
> else
> index = hash__init_new_context(mm);
> +#endif
>
> if (index < 0)
> return index;
> @@ -210,6 +218,7 @@ void __destroy_context(int context_id)
> }
> EXPORT_SYMBOL_GPL(__destroy_context);
>
> +#ifdef CONFIG_PPC_HASH_MMU
> static void destroy_contexts(mm_context_t *ctx)
> {
> int index, context_id;
> @@ -221,6 +230,7 @@ static void destroy_contexts(mm_context_t *ctx)
> }
> kfree(ctx->hash_context);
> }
> +#endif
>
> static void pmd_frag_destroy(void *pmd_frag)
> {
> @@ -273,7 +283,11 @@ void destroy_context(struct mm_struct *mm)
> process_tb[mm->context.id].prtb0 = 0;
> else
> subpage_prot_free(mm);
> +#ifdef CONFIG_PPC_HASH_MMU
> destroy_contexts(&mm->context);
> +#else
> + ida_free(&mmu_context_ida, mm->context.id);
> +#endif
> mm->context.id = MMU_NO_CONTEXT;
> }
>
> diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
> index 2bf7e1b4fd82..2f308d21700e 100644
> --- a/arch/powerpc/mm/book3s64/pgtable.c
> +++ b/arch/powerpc/mm/book3s64/pgtable.c
> @@ -11,14 +11,25 @@
> #include <asm/debugfs.h>
> #include <asm/pgalloc.h>
> #include <asm/tlb.h>
> -#include <asm/trace.h>
> #include <asm/powernv.h>
> #include <asm/firmware.h>
> #include <asm/ultravisor.h>
>
> #include <mm/mmu_decl.h>
> +#define CREATE_TRACE_POINTS
> #include <trace/events/thp.h>
>
> +struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT];
> +EXPORT_SYMBOL_GPL(mmu_psize_defs);
> +
> +int mmu_linear_psize = MMU_PAGE_4K;
> +EXPORT_SYMBOL_GPL(mmu_linear_psize);
> +int mmu_vmalloc_psize = MMU_PAGE_4K;
> +#ifdef CONFIG_SPARSEMEM_VMEMMAP
> +int mmu_vmemmap_psize = MMU_PAGE_4K;
> +#endif
> +int mmu_io_psize = MMU_PAGE_4K;
> +
> unsigned long __pmd_frag_nr;
> EXPORT_SYMBOL(__pmd_frag_nr);
> unsigned long __pmd_frag_size_shift;
> @@ -147,6 +158,7 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
> return pmd_set_protbits(__pmd(pmdv), newprot);
> }
>
> +#ifdef CONFIG_PPC_HASH_MMU
> /*
> * This is called at the end of handling a user page fault, when the
> * fault has been handled by updating a HUGE PMD entry in the linux page tables.
> @@ -159,6 +171,7 @@ void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr,
> if (radix_enabled())
> prefetch((void *)addr);
> }
> +#endif
> #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
>
> /* For use by kexec */
> @@ -220,7 +233,12 @@ static void flush_partition(unsigned int lpid, bool radix)
> "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
> /* do we need fixup here ?*/
> asm volatile("eieio; tlbsync; ptesync" : : : "memory");
> - trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
> + /*
> + * XXX: Can't use this tracepoint
> + * because this file defines the THP tracepoints
> + * trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
> + * KVM tlbie tracing has much larger deficiencies though.
> + */
> }
> }
>
> diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
> index 2a9a0cd79490..cd22557bb512 100644
> --- a/arch/powerpc/mm/book3s64/radix_pgtable.c
> +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
> @@ -315,8 +315,11 @@ static void __init radix_init_pgtable(void)
> unsigned long rts_field;
> struct memblock_region *reg;
>
> +#ifdef CONFIG_PPC_HASH_MMU
> /* We don't support slb for radix */
> mmu_slb_size = 0;
> +#endif
> +
> /*
> * Create the linear mapping, using standard page size for now
> */
> @@ -548,6 +551,7 @@ void __init radix__early_init_mmu(void)
> {
> unsigned long lpcr;
>
> +#ifdef CONFIG_PPC_HASH_MMU
> #ifdef CONFIG_PPC_64K_PAGES
> /* PAGE_SIZE mappings */
> mmu_virtual_psize = MMU_PAGE_64K;
> @@ -564,6 +568,7 @@ void __init radix__early_init_mmu(void)
> mmu_vmemmap_psize = MMU_PAGE_2M;
> } else
> mmu_vmemmap_psize = mmu_virtual_psize;
> +#endif
> #endif
> /*
> * initialize page table size
> diff --git a/arch/powerpc/mm/book3s64/slb.c b/arch/powerpc/mm/book3s64/slb.c
> index 716204aee3da..f216bbe58d15 100644
> --- a/arch/powerpc/mm/book3s64/slb.c
> +++ b/arch/powerpc/mm/book3s64/slb.c
> @@ -812,17 +812,3 @@ long do_slb_fault(struct pt_regs *regs, unsigned long ea)
> return err;
> }
> }
> -
> -void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, long err)
> -{
> - if (err == -EFAULT) {
> - if (user_mode(regs))
> - _exception(SIGSEGV, regs, SEGV_BNDERR, ea);
> - else
> - bad_page_fault(regs, ea, SIGSEGV);
> - } else if (err == -EINVAL) {
> - unrecoverable_exception(regs);
> - } else {
> - BUG();
> - }
> -}
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index beb060b96632..9fdef2216800 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -87,6 +87,7 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
> }
> EXPORT_SYMBOL_GPL(copro_handle_mm_fault);
>
> +#ifdef CONFIG_PPC_HASH_MMU
> int copro_calculate_slb(struct mm_struct *mm, u64 ea, struct copro_slb *slb)
> {
> u64 vsid, vsidkey;
> @@ -151,3 +152,4 @@ void copro_flush_all_slbs(struct mm_struct *mm)
> cxl_slbia(mm);
> }
> EXPORT_SYMBOL_GPL(copro_flush_all_slbs);
> +#endif
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index 8db0507619e2..02095e1d8e62 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -33,6 +33,7 @@
> #include <linux/hugetlb.h>
> #include <linux/uaccess.h>
>
> +#include <asm/asm-prototypes.h>
> #include <asm/firmware.h>
> #include <asm/page.h>
> #include <asm/pgtable.h>
> @@ -685,3 +686,22 @@ void bad_page_fault(struct pt_regs *regs, unsigned long address, int sig)
>
> die("Kernel access of bad area", regs, sig);
> }
> +
> +#ifdef CONFIG_PPC_BOOK3S_64
> +/*
> + * Radix takes bad slb (segment) faults as well.
> + */
> +void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, long err)
> +{
> + if (err == -EFAULT) {
> + if (user_mode(regs))
> + _exception(SIGSEGV, regs, SEGV_BNDERR, ea);
> + else
> + bad_page_fault(regs, ea, SIGSEGV);
> + } else if (err == -EINVAL) {
> + unrecoverable_exception(regs);
> + } else {
> + BUG();
> + }
> +}
> +#endif
> diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
> index 6caedc88474f..a700deae0dcc 100644
> --- a/arch/powerpc/platforms/Kconfig.cputype
> +++ b/arch/powerpc/platforms/Kconfig.cputype
> @@ -98,7 +98,6 @@ config PPC_BOOK3S_64
> select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
> select ARCH_SUPPORTS_NUMA_BALANCING
> select IRQ_WORK
> - select PPC_MM_SLICES
>
> config PPC_BOOK3E_64
> bool "Embedded processors"
> @@ -120,11 +119,13 @@ choice
> config GENERIC_CPU
> bool "Generic (POWER4 and above)"
> depends on PPC64 && !CPU_LITTLE_ENDIAN
> + select PPC_HASH_MMU
>
> config GENERIC_CPU
> bool "Generic (POWER8 and above)"
> depends on PPC64 && CPU_LITTLE_ENDIAN
> select ARCH_HAS_FAST_MULTIPLIER
> + select PPC_HASH_MMU
>
> config GENERIC_CPU
> bool "Generic 32 bits powerpc"
> @@ -133,24 +134,29 @@ config GENERIC_CPU
> config CELL_CPU
> bool "Cell Broadband Engine"
> depends on PPC_BOOK3S_64 && !CPU_LITTLE_ENDIAN
> + select PPC_HASH_MMU
>
> config POWER5_CPU
> bool "POWER5"
> depends on PPC_BOOK3S_64 && !CPU_LITTLE_ENDIAN
> + select PPC_HASH_MMU
>
> config POWER6_CPU
> bool "POWER6"
> depends on PPC_BOOK3S_64 && !CPU_LITTLE_ENDIAN
> + select PPC_HASH_MMU
>
> config POWER7_CPU
> bool "POWER7"
> depends on PPC_BOOK3S_64
> select ARCH_HAS_FAST_MULTIPLIER
> + select PPC_HASH_MMU
>
> config POWER8_CPU
> bool "POWER8"
> depends on PPC_BOOK3S_64
> select ARCH_HAS_FAST_MULTIPLIER
> + select PPC_HASH_MMU
>
> config POWER9_CPU
> bool "POWER9"
> @@ -358,9 +364,21 @@ config PPC_RADIX_MMU
> is only implemented by IBM Power9 CPUs, if you don't have one of them
> you can probably disable this.
>
> +config PPC_HASH_MMU
> + bool "Hash MMU Support"
> + depends on PPC_BOOK3S_64
> + default y
> + select PPC_MM_SLICES
> + help
> + Enable support for the Power ISA Hash style MMU. This is implemented
> + by all IBM Power and other Book3S CPUs.
> +
> + If you're unsure, say Y.
> +
> config PPC_RADIX_MMU_DEFAULT
> bool "Default to using the Radix MMU when possible"
> depends on PPC_RADIX_MMU
> + depends on PPC_HASH_MMU
> default y
> help
> When the hardware supports the Radix MMU, default to using it unless
> diff --git a/arch/powerpc/platforms/powernv/idle.c b/arch/powerpc/platforms/powernv/idle.c
> index 78599bca66c2..3f2633b46ca3 100644
> --- a/arch/powerpc/platforms/powernv/idle.c
> +++ b/arch/powerpc/platforms/powernv/idle.c
> @@ -491,12 +491,14 @@ static unsigned long power7_idle_insn(unsigned long type)
>
> mtspr(SPRN_SPRG3, local_paca->sprg_vdso);
>
> +#ifdef CONFIG_PPC_HASH_MMU
> /*
> * The SLB has to be restored here, but it sometimes still
> * contains entries, so the __ variant must be used to prevent
> * multi hits.
> */
> __slb_restore_bolted_realmode();
> +#endif
>
> return srr1;
> }
> diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
> index 11fdae81b5dd..fb7061ca63d0 100644
> --- a/arch/powerpc/platforms/powernv/setup.c
> +++ b/arch/powerpc/platforms/powernv/setup.c
> @@ -168,6 +168,7 @@ static void __init pnv_init(void)
> #endif
> add_preferred_console("hvc", 0, NULL);
>
> +#ifdef CONFIG_PPC_HASH_MMU
> if (!radix_enabled()) {
> int i;
>
> @@ -175,6 +176,7 @@ static void __init pnv_init(void)
> for_each_possible_cpu(i)
> paca_ptrs[i]->mce_faulty_slbs = memblock_alloc_node(mmu_slb_size, __alignof__(*paca_ptrs[i]->mce_faulty_slbs), cpu_to_node(i));
> }
> +#endif
> }
>
> static void __init pnv_init_IRQ(void)
> diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
> index 0ec9640335bb..0146bfb9ee2a 100644
> --- a/arch/powerpc/xmon/xmon.c
> +++ b/arch/powerpc/xmon/xmon.c
> @@ -1119,9 +1119,11 @@ cmds(struct pt_regs *excp)
> show_tasks();
> break;
> #ifdef CONFIG_PPC_BOOK3S
> +#ifdef CONFIG_PPC_HASH_MMU
> case 'u':
> dump_segments();
> break;
> +#endif
> #elif defined(CONFIG_44x)
> case 'u':
> dump_tlb_44x();
> @@ -2419,7 +2421,7 @@ static void dump_tracing(void)
> static void dump_one_paca(int cpu)
> {
> struct paca_struct *p;
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#if defined(CONFIG_PPC_BOOK3S_64) && defined(CONFIG_PPC_HASH_MMU)
> int i = 0;
> #endif
>
> @@ -2461,6 +2463,7 @@ static void dump_one_paca(int cpu)
> DUMP(p, cpu_start, "%#-*x");
> DUMP(p, kexec_state, "%#-*x");
> #ifdef CONFIG_PPC_BOOK3S_64
> +#ifdef CONFIG_PPC_HASH_MMU
> if (!early_radix_enabled()) {
> for (i = 0; i < SLB_NUM_BOLTED; i++) {
> u64 esid, vsid;
> @@ -2488,6 +2491,7 @@ static void dump_one_paca(int cpu)
> 22, "slb_cache", i, p->slb_cache[i]);
> }
> }
> +#endif
>
> DUMP(p, rfi_flush_fallback_area, "%-*px");
> #endif
> @@ -3570,7 +3574,7 @@ static void xmon_print_symbol(unsigned long address, const char *mid,
> printf("%s", after);
> }
>
> -#ifdef CONFIG_PPC_BOOK3S_64
> +#if defined(CONFIG_PPC_BOOK3S_64) && defined(CONFIG_PPC_HASH_MMU)
> void dump_segments(void)
> {
> int i;
> --
> 2.23.0
^ permalink raw reply
* Re: [PATCH v4] powerpc: setup_64: set up PACA before parsing device tree
From: Michael Ellerman @ 2020-03-19 4:19 UTC (permalink / raw)
To: Daniel Axtens, linuxppc-dev, npiggin; +Cc: ajd, Daniel Axtens
In-Reply-To: <20200316042227.12445-1-dja@axtens.net>
Daniel Axtens <dja@axtens.net> writes:
> Currently, we set up the PACA after parsing the device tree for CPU
> features. Before that, r13 contains random data, which means there is
> random data in r13 while we're running the generic dt parsing code.
>
> This random data varies depending on whether we boot through a vmlinux or a
> zImage: for the vmlinux case it's usually around zero, but for zImages we
> see random values like 912a72603d420015.
>
> This is poor practice, and can also lead to difficult-to-debug crashes. For
> example, when kcov is enabled, the kcov instrumentation attempts to read
> preempt_count out of the current task, which goes via the PACA. This then
> crashes in the zImage case.
>
> To resolve this:
>
> - move the PACA setup to before the cpu feature parsing.
>
> - because we no longer have access to cpu feature flags in PACA setup,
> change the HV feature test in the PACA setup path to consider the actual
> value of the MSR rather than the CPU feature.
>
> Translations get switched on once we leave early_setup, so I think we'd
> already catch any other cases where the PACA or task aren't set up.
>
> Boot tested on a P9 guest and host.
>
> Fixes: fb0b0a73b223 ("powerpc: Enable kcov")
> Cc: Andrew Donnellan <ajd@linux.ibm.com>
> Cc: stable@vger.kernel.org
> Reviewed-by: Andrew Donnellan <ajd@linux.ibm.com>
> Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
> Signed-off-by: Daniel Axtens <dja@axtens.net>
>
> ---
>
> Regarding moving the comment about printk()-safety:
> I am about 75% sure that the thing that makes printk() safe is the PACA,
> not the CPU features. That's what commit 24d9649574fb ("[POWERPC] Document
> when printk is useable") seems to indicate, but as someone wise recently
> told me, "bootstrapping is hard", so I may be totally wrong.
Yeah:
When debugging early boot problems, it's common to sprinkle printk's
all over the place. However, on 64-bit powerpc, this can lead to
memory corruption if done too early due to the PACA pointer and
lockdep core not being initialized.
ie. printk was calling into lockdep and that was blowing chunks. My
recollection is that lockdep was crashing because it wasn't initialised
rather than anything to do with the paca, but I could be wrong.
Presumably the lockdep issue was actually fixed in:
06bea3dbfe6a ("locking/lockdep: Eliminate lockdep_init()") (Feb 2016)
Which removed the lockdep_init() call entirely.
And that's why we haven't noticed any issues despite the fact that
dt_cpu_ftrs_init() *is* calling printk:
13190702: (13190702): [ 0.013183435,5] INIT: Starting kernel at 0x20010000, fdt at 0x3054c230 15395 bytes
41352186: (41352186): [ 0.000000][ T0] dt-cpu-ftrs: setup for ISA 3000
41479101: (41479101): [ 0.000000][ T0] dt-cpu-ftrs: not enabling: system-call-vectored (disabled or unsupported by kernel)
41718546: (41718546): [ 0.000000][ T0] dt-cpu-ftrs: final cpu/mmu features = 0x0001986f8f5fb1a7 0x3c006041
41923251: (41923251): [ 0.000000][ T0] radix-mmu: Page sizes from device-tree:
As you've seen I pulled on this string and found problems with stack
protector. But I didn't actually see any problems with printk.
So possibly we should update that comment to not refer to printk but
instead say something like "don't call generic code".
cheers
> v4: Update commit message and clarify that the mfmsr() approach is not
> for general use. Thanks Nick Piggin.
>
> v3: Update comment, thanks Christophe Leroy.
> Remove a comment in dt_cpu_ftrs.c that is no longer accurate - thanks
> Andrew. I think we want to retain all the code still, but I'm open to
> being told otherwise.
> ---
> arch/powerpc/kernel/dt_cpu_ftrs.c | 1 -
> arch/powerpc/kernel/paca.c | 10 +++++++---
> arch/powerpc/kernel/setup_64.c | 20 +++++++++++++++-----
> 3 files changed, 22 insertions(+), 9 deletions(-)
>
> diff --git a/arch/powerpc/kernel/dt_cpu_ftrs.c b/arch/powerpc/kernel/dt_cpu_ftrs.c
> index 182b4047c1ef..36bc0d5c4f3a 100644
> --- a/arch/powerpc/kernel/dt_cpu_ftrs.c
> +++ b/arch/powerpc/kernel/dt_cpu_ftrs.c
> @@ -139,7 +139,6 @@ static void __init cpufeatures_setup_cpu(void)
> /* Initialize the base environment -- clear FSCR/HFSCR. */
> hv_mode = !!(mfmsr() & MSR_HV);
> if (hv_mode) {
> - /* CPU_FTR_HVMODE is used early in PACA setup */
> cur_cpu_spec->cpu_features |= CPU_FTR_HVMODE;
> mtspr(SPRN_HFSCR, 0);
> }
> diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
> index 949eceb254d8..0ee6308541b1 100644
> --- a/arch/powerpc/kernel/paca.c
> +++ b/arch/powerpc/kernel/paca.c
> @@ -214,11 +214,15 @@ void setup_paca(struct paca_struct *new_paca)
> /* On Book3E, initialize the TLB miss exception frames */
> mtspr(SPRN_SPRG_TLB_EXFRAME, local_paca->extlb);
> #else
> - /* In HV mode, we setup both HPACA and PACA to avoid problems
> + /*
> + * In HV mode, we setup both HPACA and PACA to avoid problems
> * if we do a GET_PACA() before the feature fixups have been
> - * applied
> + * applied.
> + *
> + * Normally you should test against CPU_FTR_HVMODE, but CPU features
> + * are not yet set up when we first reach here.
> */
> - if (early_cpu_has_feature(CPU_FTR_HVMODE))
> + if (mfmsr() & MSR_HV)
> mtspr(SPRN_SPRG_HPACA, local_paca);
> #endif
> mtspr(SPRN_SPRG_PACA, local_paca);
> diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
> index e05e6dd67ae6..2259da8e8685 100644
> --- a/arch/powerpc/kernel/setup_64.c
> +++ b/arch/powerpc/kernel/setup_64.c
> @@ -285,18 +285,28 @@ void __init early_setup(unsigned long dt_ptr)
>
> /* -------- printk is _NOT_ safe to use here ! ------- */
>
> - /* Try new device tree based feature discovery ... */
> - if (!dt_cpu_ftrs_init(__va(dt_ptr)))
> - /* Otherwise use the old style CPU table */
> - identify_cpu(0, mfspr(SPRN_PVR));
> + /*
> + * Assume we're on cpu 0 for now.
> + *
> + * We need to load a PACA very early if we are using kcov. kcov will
> + * call in_task() in its instrumentation, which relies on the current
> + * task from the PACA. dt_cpu_ftrs_init is coveraged-enabled and also
> + * calls into the coverage-enabled generic dt library.
> + *
> + * Set up a temporary paca. It is going to be replaced below.
> + */
>
> - /* Assume we're on cpu 0 for now. Don't write to the paca yet! */
> initialise_paca(&boot_paca, 0);
> setup_paca(&boot_paca);
> fixup_boot_paca();
>
> /* -------- printk is now safe to use ------- */
>
> + /* Try new device tree based feature discovery ... */
> + if (!dt_cpu_ftrs_init(__va(dt_ptr)))
> + /* Otherwise use the old style CPU table */
> + identify_cpu(0, mfspr(SPRN_PVR));
> +
> /* Enable early debugging if any specified (see udbg.h) */
> udbg_early_init();
>
> --
> 2.20.1
^ permalink raw reply
* [PATCH v8 00/14] powerpc/vas: Page fault handling for user space NX requests
From: Haren Myneni @ 2020-03-19 6:08 UTC (permalink / raw)
To: mpe, linuxppc-dev; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, ajd
On power9, Virtual Accelerator Switchboard (VAS) allows user space or
kernel to communicate with Nest Accelerator (NX) directly using COPY/PASTE
instructions. NX provides various functionalities such as compression,
encryption and etc. But only compression (842 and GZIP formats) is
supported in Linux kernel on power9.
842 compression driver (drivers/crypto/nx/nx-842-powernv.c)
is already included in Linux. Only GZIP support will be available from
user space.
Applications can issue GZIP compression / decompression requests to NX with
COPY/PASTE instructions. When NX is processing these requests, can hit
fault on the request buffer (not in memory). It issues an interrupt and
pastes fault CRB in fault FIFO. Expects kernel to handle this fault and
return credits for both send and fault windows after processing.
This patch series adds IRQ and fault window setup, and NX fault handling:
- Alloc IRQ and trigger port address, and configure IRQ per VAS instance.
- Set port# for each window to generate an interrupt when noticed fault.
- Set fault window and FIFO on which NX paste fault CRB.
- Setup IRQ thread fault handler per VAS instance.
- When receiving an interrupt, Read CRBs from fault FIFO and update
coprocessor_status_block (CSB) in the corresponding CRB with translation
failure (CSB_CC_TRANSLATION). After issuing NX requests, process polls
on CSB address. When it sees translation error, can touch the request
buffer to bring the page in to memory and reissue NX request.
- If copy_to_user fails on user space CSB address, OS sends SEGV signal.
Tested these patches with NX-GZIP support and will be posting this series
soon.
Patches 1 & 2: Define alloc IRQ and get port address per chip which are needed
to alloc IRQ per VAS instance.
Patch 3: Define nx_fault_stamp on which NX writes fault status for the fault
CRB
Patch 4: Alloc and setup IRQ and trigger port address for each VAS instance
Patch 5: Setup fault window per each VAS instance. This window is used for
NX to paste fault CRB in FIFO.
Patches 6 & 7: Setup threaded IRQ per VAS and register NX with fault window
ID and port number for each send window so that NX paste fault CRB
in this window.
Patch 8: Reference to pid and mm so that pid is not used until window closed.
Needed for multi thread application where child can open a window
and can be used by parent later.
Patches 9 and 10: Process CRBs from fault FIFO and notify tasks by
updating CSB or through signals.
Patches 11 and 12: Return credits for send and fault windows after handling
faults.
Patch 14:Fix closing send window after all credits are returned. This issue
happens only for user space requests. No page faults on kernel
request buffer.
Changelog:
V2:
- Use threaded IRQ instead of own kernel thread handler
- Use pswid instead of user space CSB address to find valid CRB
- Removed unused macros and other changes as suggested by Christoph Hellwig
V3:
- Rebased to 5.5-rc2
- Use struct pid * instead of pid_t for vas_window tgid
- Code cleanup as suggested by Christoph Hellwig
V4:
- Define xive alloc and get IRQ info based on chip ID and use these
functions for IRQ setup per VAS instance. It eliminates skiboot
dependency as suggested by Oliver.
V5:
- Do not update CSB if the process is exiting (patch9)
V6:
- Add interrupt handler instead of default one and return IRQ_HANDLED
if the fault handling thread is already in progress. (Patch6)
- Use platform send window ID and CCW[0] bit to find valid CRB in
fault FIFO (Patch6).
- Return fault address to user space in BE and other changes as
suggested by Michael Neuling. (patch9)
- Rebased to 5.6-rc4
V7:
- Fix sparse warnings (patches 6,9 and 10)
V8:
- Move mm_context_remove_copro() before mmdrop() (patch8)
- Move barrier before csb.flags store and add WARN_ON_ONCE() checks (patch9)
Haren Myneni (14):
powerpc/xive: Define xive_native_alloc_irq_on_chip()
powerpc/xive: Define xive_native_alloc_get_irq_info()
powerpc/vas: Define nx_fault_stamp in coprocessor_request_block
powerpc/vas: Alloc and setup IRQ and trigger port address
powerpc/vas: Setup fault window per VAS instance
powerpc/vas: Setup thread IRQ handler per VAS instance
powerpc/vas: Register NX with fault window ID and IRQ port value
powerpc/vas: Take reference to PID and mm for user space windows
powerpc/vas: Update CSB and notify process for fault CRBs
powerpc/vas: Print CRB and FIFO values
powerpc/vas: Do not use default credits for receive window
powerpc/vas: Return credits after handling fault
powerpc/vas: Display process stuck message
powerpc/vas: Free send window in VAS instance after credits returned
arch/powerpc/include/asm/icswx.h | 18 +-
arch/powerpc/include/asm/xive.h | 11 +-
arch/powerpc/platforms/powernv/Makefile | 2 +-
arch/powerpc/platforms/powernv/ocxl.c | 20 +-
arch/powerpc/platforms/powernv/vas-debug.c | 2 +-
arch/powerpc/platforms/powernv/vas-fault.c | 332 ++++++++++++++++++++++++++++
arch/powerpc/platforms/powernv/vas-window.c | 185 ++++++++++++++--
arch/powerpc/platforms/powernv/vas.c | 101 ++++++++-
arch/powerpc/platforms/powernv/vas.h | 51 ++++-
arch/powerpc/sysdev/xive/native.c | 29 ++-
10 files changed, 704 insertions(+), 47 deletions(-)
create mode 100644 arch/powerpc/platforms/powernv/vas-fault.c
--
1.8.3.1
^ permalink raw reply
* [PATCH v8 01/14] powerpc/xive: Define xive_native_alloc_irq_on_chip()
From: Haren Myneni @ 2020-03-19 6:12 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
This function allocates IRQ on a specific chip. VAS needs per chip
IRQ allocation and will have IRQ handler per VAS instance.
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/include/asm/xive.h | 9 ++++++++-
arch/powerpc/sysdev/xive/native.c | 6 +++---
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 93f982db..d08ea11 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -5,6 +5,8 @@
#ifndef _ASM_POWERPC_XIVE_H
#define _ASM_POWERPC_XIVE_H
+#include <asm/opal-api.h>
+
#define XIVE_INVALID_VP 0xffffffff
#ifdef CONFIG_PPC_XIVE
@@ -108,7 +110,6 @@ struct xive_q {
int xive_native_populate_irq_data(u32 hw_irq,
struct xive_irq_data *data);
void xive_cleanup_irq_data(struct xive_irq_data *xd);
-u32 xive_native_alloc_irq(void);
void xive_native_free_irq(u32 irq);
int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
@@ -137,6 +138,12 @@ int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
u32 qindex);
int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
bool xive_native_has_queue_state_support(void);
+extern u32 xive_native_alloc_irq_on_chip(u32 chip_id);
+
+static inline u32 xive_native_alloc_irq(void)
+{
+ return xive_native_alloc_irq_on_chip(OPAL_XIVE_ANY_CHIP);
+}
#else
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 0ff6b73..14d4406 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -279,12 +279,12 @@ static int xive_native_get_ipi(unsigned int cpu, struct xive_cpu *xc)
}
#endif /* CONFIG_SMP */
-u32 xive_native_alloc_irq(void)
+u32 xive_native_alloc_irq_on_chip(u32 chip_id)
{
s64 rc;
for (;;) {
- rc = opal_xive_allocate_irq(OPAL_XIVE_ANY_CHIP);
+ rc = opal_xive_allocate_irq(chip_id);
if (rc != OPAL_BUSY)
break;
msleep(OPAL_BUSY_DELAY_MS);
@@ -293,7 +293,7 @@ u32 xive_native_alloc_irq(void)
return 0;
return rc;
}
-EXPORT_SYMBOL_GPL(xive_native_alloc_irq);
+EXPORT_SYMBOL_GPL(xive_native_alloc_irq_on_chip);
void xive_native_free_irq(u32 irq)
{
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 02/14] powerpc/xive: Define xive_native_alloc_get_irq_info()
From: Haren Myneni @ 2020-03-19 6:13 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
pnv_ocxl_alloc_xive_irq() in ocxl.c allocates IRQ and gets trigger port
address. VAS also needs this function, but based on chip ID. So moved
this common function to xive/native.c.
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/include/asm/xive.h | 2 ++
arch/powerpc/platforms/powernv/ocxl.c | 20 ++------------------
arch/powerpc/sysdev/xive/native.c | 23 +++++++++++++++++++++++
3 files changed, 27 insertions(+), 18 deletions(-)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index d08ea11..fd337da 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -139,6 +139,8 @@ int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
bool xive_native_has_queue_state_support(void);
extern u32 xive_native_alloc_irq_on_chip(u32 chip_id);
+extern int xive_native_alloc_get_irq_info(u32 chip_id, u32 *irq,
+ u64 *trigger_addr);
static inline u32 xive_native_alloc_irq(void)
{
diff --git a/arch/powerpc/platforms/powernv/ocxl.c b/arch/powerpc/platforms/powernv/ocxl.c
index 8c65aac..fb8f99a 100644
--- a/arch/powerpc/platforms/powernv/ocxl.c
+++ b/arch/powerpc/platforms/powernv/ocxl.c
@@ -487,24 +487,8 @@ int pnv_ocxl_spa_remove_pe_from_cache(void *platform_data, int pe_handle)
int pnv_ocxl_alloc_xive_irq(u32 *irq, u64 *trigger_addr)
{
- __be64 flags, trigger_page;
- s64 rc;
- u32 hwirq;
-
- hwirq = xive_native_alloc_irq();
- if (!hwirq)
- return -ENOENT;
-
- rc = opal_xive_get_irq_info(hwirq, &flags, NULL, &trigger_page, NULL,
- NULL);
- if (rc || !trigger_page) {
- xive_native_free_irq(hwirq);
- return -ENOENT;
- }
- *irq = hwirq;
- *trigger_addr = be64_to_cpu(trigger_page);
- return 0;
-
+ return xive_native_alloc_get_irq_info(OPAL_XIVE_ANY_CHIP, irq,
+ trigger_addr);
}
EXPORT_SYMBOL_GPL(pnv_ocxl_alloc_xive_irq);
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 14d4406..abdd892 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -295,6 +295,29 @@ u32 xive_native_alloc_irq_on_chip(u32 chip_id)
}
EXPORT_SYMBOL_GPL(xive_native_alloc_irq_on_chip);
+int xive_native_alloc_get_irq_info(u32 chip_id, u32 *irq, u64 *trigger_addr)
+{
+ __be64 flags, trigger_page;
+ u32 hwirq;
+ s64 rc;
+
+ hwirq = xive_native_alloc_irq_on_chip(chip_id);
+ if (!hwirq)
+ return -ENOENT;
+
+ rc = opal_xive_get_irq_info(hwirq, &flags, NULL, &trigger_page, NULL,
+ NULL);
+ if (rc || !trigger_page) {
+ xive_native_free_irq(hwirq);
+ return -ENOENT;
+ }
+ *irq = hwirq;
+ *trigger_addr = be64_to_cpu(trigger_page);
+
+ return 0;
+}
+EXPORT_SYMBOL(xive_native_alloc_get_irq_info);
+
void xive_native_free_irq(u32 irq)
{
for (;;) {
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 03/14] powerpc/vas: Define nx_fault_stamp in coprocessor_request_block
From: Haren Myneni @ 2020-03-19 6:13 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
Kernel sets fault address and status in CRB for NX page fault on user
space address after processing page fault. User space gets the signal
and handles the fault mentioned in CRB by bringing the page in to
memory and send NX request again.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/include/asm/icswx.h | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/icswx.h b/arch/powerpc/include/asm/icswx.h
index 9872f85..b233d1e 100644
--- a/arch/powerpc/include/asm/icswx.h
+++ b/arch/powerpc/include/asm/icswx.h
@@ -108,6 +108,17 @@ struct data_descriptor_entry {
__be64 address;
} __packed __aligned(DDE_ALIGN);
+/* 4.3.2 NX-stamped Fault CRB */
+
+#define NX_STAMP_ALIGN (0x10)
+
+struct nx_fault_stamp {
+ __be64 fault_storage_addr;
+ __be16 reserved;
+ __u8 flags;
+ __u8 fault_status;
+ __be32 pswid;
+} __packed __aligned(NX_STAMP_ALIGN);
/* Chapter 6.5.2 Coprocessor-Request Block (CRB) */
@@ -135,7 +146,12 @@ struct coprocessor_request_block {
struct coprocessor_completion_block ccb;
- u8 reserved[48];
+ union {
+ struct nx_fault_stamp nx;
+ u8 reserved[16];
+ } stamp;
+
+ u8 reserved[32];
struct coprocessor_status_block csb;
} __packed __aligned(CRB_ALIGN);
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 04/14] powerpc/vas: Alloc and setup IRQ and trigger port address
From: Haren Myneni @ 2020-03-19 6:14 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
Alloc IRQ and get trigger port address for each VAS instance. Kernel
register this IRQ per VAS instance and sets this port for each send
window. NX interrupts the kernel when it sees page fault.
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas.c | 34 ++++++++++++++++++++++++++++------
arch/powerpc/platforms/powernv/vas.h | 2 ++
2 files changed, 30 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/vas.c b/arch/powerpc/platforms/powernv/vas.c
index ed9cc6d..168ab68 100644
--- a/arch/powerpc/platforms/powernv/vas.c
+++ b/arch/powerpc/platforms/powernv/vas.c
@@ -15,6 +15,7 @@
#include <linux/of_address.h>
#include <linux/of.h>
#include <asm/prom.h>
+#include <asm/xive.h>
#include "vas.h"
@@ -25,10 +26,12 @@
static int init_vas_instance(struct platform_device *pdev)
{
- int rc, cpu, vasid;
- struct resource *res;
- struct vas_instance *vinst;
struct device_node *dn = pdev->dev.of_node;
+ struct vas_instance *vinst;
+ uint32_t chipid, irq;
+ struct resource *res;
+ int rc, cpu, vasid;
+ uint64_t port;
rc = of_property_read_u32(dn, "ibm,vas-id", &vasid);
if (rc) {
@@ -36,6 +39,12 @@ static int init_vas_instance(struct platform_device *pdev)
return -ENODEV;
}
+ rc = of_property_read_u32(dn, "ibm,chip-id", &chipid);
+ if (rc) {
+ pr_err("No ibm,chip-id property for %s?\n", pdev->name);
+ return -ENODEV;
+ }
+
if (pdev->num_resources != 4) {
pr_err("Unexpected DT configuration for [%s, %d]\n",
pdev->name, vasid);
@@ -69,9 +78,22 @@ static int init_vas_instance(struct platform_device *pdev)
vinst->paste_win_id_shift = 63 - res->end;
- pr_devel("Initialized instance [%s, %d], paste_base 0x%llx, "
- "paste_win_id_shift 0x%llx\n", pdev->name, vasid,
- vinst->paste_base_addr, vinst->paste_win_id_shift);
+ rc = xive_native_alloc_get_irq_info(chipid, &irq, &port);
+ if (rc)
+ return rc;
+
+ vinst->virq = irq_create_mapping(NULL, irq);
+ if (!vinst->virq) {
+ pr_err("Inst%d: Unable to map global irq %d\n",
+ vinst->vas_id, irq);
+ return -EINVAL;
+ }
+
+ vinst->irq_port = port;
+ pr_devel("Initialized instance [%s, %d] paste_base 0x%llx paste_win_id_shift 0x%llx IRQ %d Port 0x%llx\n",
+ pdev->name, vasid, vinst->paste_base_addr,
+ vinst->paste_win_id_shift, vinst->virq,
+ vinst->irq_port);
for_each_possible_cpu(cpu) {
if (cpu_to_chip_id(cpu) == of_get_ibm_chip_id(dn))
diff --git a/arch/powerpc/platforms/powernv/vas.h b/arch/powerpc/platforms/powernv/vas.h
index 5574aec..598608b 100644
--- a/arch/powerpc/platforms/powernv/vas.h
+++ b/arch/powerpc/platforms/powernv/vas.h
@@ -313,6 +313,8 @@ struct vas_instance {
u64 paste_base_addr;
u64 paste_win_id_shift;
+ u64 irq_port;
+ int virq;
struct mutex mutex;
struct vas_window *rxwin[VAS_COP_TYPE_MAX];
struct vas_window *windows[VAS_WINDOWS_PER_CHIP];
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 05/14] powerpc/vas: Setup fault window per VAS instance
From: Haren Myneni @ 2020-03-19 6:15 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
Setup fault window for each VAS instance. When NX gets a fault on
request buffer, write fault CRBs in the corresponding fault FIFO and
then sends an interrupt to the OS.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/Makefile | 2 +-
arch/powerpc/platforms/powernv/vas-fault.c | 77 +++++++++++++++++++++++++++++
arch/powerpc/platforms/powernv/vas-window.c | 4 +-
arch/powerpc/platforms/powernv/vas.c | 20 ++++++++
arch/powerpc/platforms/powernv/vas.h | 16 ++++++
5 files changed, 116 insertions(+), 3 deletions(-)
create mode 100644 arch/powerpc/platforms/powernv/vas-fault.c
diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
index c0f8120..395789f 100644
--- a/arch/powerpc/platforms/powernv/Makefile
+++ b/arch/powerpc/platforms/powernv/Makefile
@@ -17,7 +17,7 @@ obj-$(CONFIG_MEMORY_FAILURE) += opal-memory-errors.o
obj-$(CONFIG_OPAL_PRD) += opal-prd.o
obj-$(CONFIG_PERF_EVENTS) += opal-imc.o
obj-$(CONFIG_PPC_MEMTRACE) += memtrace.o
-obj-$(CONFIG_PPC_VAS) += vas.o vas-window.o vas-debug.o
+obj-$(CONFIG_PPC_VAS) += vas.o vas-window.o vas-debug.o vas-fault.o
obj-$(CONFIG_OCXL_BASE) += ocxl.o
obj-$(CONFIG_SCOM_DEBUGFS) += opal-xscom.o
obj-$(CONFIG_PPC_SECURE_BOOT) += opal-secvar.o
diff --git a/arch/powerpc/platforms/powernv/vas-fault.c b/arch/powerpc/platforms/powernv/vas-fault.c
new file mode 100644
index 0000000..4044998
--- /dev/null
+++ b/arch/powerpc/platforms/powernv/vas-fault.c
@@ -0,0 +1,77 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * VAS Fault handling.
+ * Copyright 2019, IBM Corporation
+ */
+
+#define pr_fmt(fmt) "vas: " fmt
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+#include <linux/kthread.h>
+#include <asm/icswx.h>
+
+#include "vas.h"
+
+/*
+ * The maximum FIFO size for fault window can be 8MB
+ * (VAS_RX_FIFO_SIZE_MAX). Using 4MB FIFO since each VAS
+ * instance will be having fault window.
+ * 8MB FIFO can be used if expects more faults for each VAS
+ * instance.
+ */
+#define VAS_FAULT_WIN_FIFO_SIZE (4 << 20)
+
+/*
+ * Fault window is opened per VAS instance. NX pastes fault CRB in fault
+ * FIFO upon page faults.
+ */
+int vas_setup_fault_window(struct vas_instance *vinst)
+{
+ struct vas_rx_win_attr attr;
+
+ vinst->fault_fifo_size = VAS_FAULT_WIN_FIFO_SIZE;
+ vinst->fault_fifo = kzalloc(vinst->fault_fifo_size, GFP_KERNEL);
+ if (!vinst->fault_fifo) {
+ pr_err("Unable to alloc %d bytes for fault_fifo\n",
+ vinst->fault_fifo_size);
+ return -ENOMEM;
+ }
+
+ /*
+ * Invalidate all CRB entries. NX pastes valid entry for each fault.
+ */
+ memset(vinst->fault_fifo, FIFO_INVALID_ENTRY, vinst->fault_fifo_size);
+ vas_init_rx_win_attr(&attr, VAS_COP_TYPE_FAULT);
+
+ attr.rx_fifo_size = vinst->fault_fifo_size;
+ attr.rx_fifo = vinst->fault_fifo;
+
+ /*
+ * Max creds is based on number of CRBs can fit in the FIFO.
+ * (fault_fifo_size/CRB_SIZE). If 8MB FIFO is used, max creds
+ * will be 0xffff since the receive creds field is 16bits wide.
+ */
+ attr.wcreds_max = vinst->fault_fifo_size / CRB_SIZE;
+ attr.lnotify_lpid = 0;
+ attr.lnotify_pid = mfspr(SPRN_PID);
+ attr.lnotify_tid = mfspr(SPRN_PID);
+
+ vinst->fault_win = vas_rx_win_open(vinst->vas_id, VAS_COP_TYPE_FAULT,
+ &attr);
+
+ if (IS_ERR(vinst->fault_win)) {
+ pr_err("VAS: Error %ld opening FaultWin\n",
+ PTR_ERR(vinst->fault_win));
+ kfree(vinst->fault_fifo);
+ return PTR_ERR(vinst->fault_win);
+ }
+
+ pr_devel("VAS: Created FaultWin %d, LPID/PID/TID [%d/%d/%d]\n",
+ vinst->fault_win->winid, attr.lnotify_lpid,
+ attr.lnotify_pid, attr.lnotify_tid);
+
+ return 0;
+}
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index 0c0d27d..1783fa9 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -827,9 +827,9 @@ void vas_init_rx_win_attr(struct vas_rx_win_attr *rxattr, enum vas_cop_type cop)
rxattr->fault_win = true;
rxattr->notify_disable = true;
rxattr->rx_wcred_mode = true;
- rxattr->tx_wcred_mode = true;
rxattr->rx_win_ord_mode = true;
- rxattr->tx_win_ord_mode = true;
+ rxattr->rej_no_credit = true;
+ rxattr->tc_mode = VAS_THRESH_DISABLED;
} else if (cop == VAS_COP_TYPE_FTW) {
rxattr->user_win = true;
rxattr->intr_disable = true;
diff --git a/arch/powerpc/platforms/powernv/vas.c b/arch/powerpc/platforms/powernv/vas.c
index 168ab68..557c8e4 100644
--- a/arch/powerpc/platforms/powernv/vas.c
+++ b/arch/powerpc/platforms/powernv/vas.c
@@ -24,6 +24,11 @@
static DEFINE_PER_CPU(int, cpu_vas_id);
+static int vas_irq_fault_window_setup(struct vas_instance *vinst)
+{
+ return vas_setup_fault_window(vinst);
+}
+
static int init_vas_instance(struct platform_device *pdev)
{
struct device_node *dn = pdev->dev.of_node;
@@ -104,6 +109,21 @@ static int init_vas_instance(struct platform_device *pdev)
list_add(&vinst->node, &vas_instances);
mutex_unlock(&vas_mutex);
+ /*
+ * IRQ and fault handling setup is needed only for user space
+ * send windows.
+ */
+ if (vinst->virq) {
+ rc = vas_irq_fault_window_setup(vinst);
+ /*
+ * Fault window is used only for user space send windows.
+ * So if vinst->virq is NULL, tx_win_open returns -ENODEV
+ * for user space.
+ */
+ if (rc)
+ vinst->virq = 0;
+ }
+
vas_instance_init_dbgdir(vinst);
dev_set_drvdata(&pdev->dev, vinst);
diff --git a/arch/powerpc/platforms/powernv/vas.h b/arch/powerpc/platforms/powernv/vas.h
index 598608b..6c4baf5 100644
--- a/arch/powerpc/platforms/powernv/vas.h
+++ b/arch/powerpc/platforms/powernv/vas.h
@@ -296,6 +296,17 @@ enum vas_notify_after_count {
};
/*
+ * NX can generate an interrupt for multiple faults and expects kernel
+ * to process all of them. So read all valid CRB entries until find the
+ * invalid one. So use pswid which is pasted by NX and ccw[0] (reserved
+ * bit in BE) to check valid CRB.
+ * Invalidate FIFO during allocation and process all entries from last
+ * successful read until finds invalid pswid and ccw[0] values.
+ */
+#define FIFO_INVALID_ENTRY 0xffffffff
+#define CCW0_INVALID 1
+
+/*
* One per instance of VAS. Each instance will have a separate set of
* receive windows, one per coprocessor type.
*
@@ -315,6 +326,10 @@ struct vas_instance {
u64 irq_port;
int virq;
+ int fault_fifo_size;
+ void *fault_fifo;
+ struct vas_window *fault_win; /* Fault window */
+
struct mutex mutex;
struct vas_window *rxwin[VAS_COP_TYPE_MAX];
struct vas_window *windows[VAS_WINDOWS_PER_CHIP];
@@ -408,6 +423,7 @@ struct vas_winctx {
extern void vas_instance_init_dbgdir(struct vas_instance *vinst);
extern void vas_window_init_dbgdir(struct vas_window *win);
extern void vas_window_free_dbgdir(struct vas_window *win);
+extern int vas_setup_fault_window(struct vas_instance *vinst);
static inline void vas_log_write(struct vas_window *win, char *name,
void *regptr, u64 val)
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 06/14] powerpc/vas: Setup thread IRQ handler per VAS instance
From: Haren Myneni @ 2020-03-19 6:15 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
Setup thread IRQ handler per each VAS instance. When NX sees a fault
on CRB, kernel gets an interrupt and vas_fault_handler will be
executed to process fault CRBs. Read all valid CRBs from fault FIFO,
determine the corresponding send window from CRB and process fault
requests.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-fault.c | 90 +++++++++++++++++++++++++++++
arch/powerpc/platforms/powernv/vas-window.c | 60 +++++++++++++++++++
arch/powerpc/platforms/powernv/vas.c | 49 +++++++++++++++-
arch/powerpc/platforms/powernv/vas.h | 6 ++
4 files changed, 204 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/platforms/powernv/vas-fault.c b/arch/powerpc/platforms/powernv/vas-fault.c
index 4044998..1c6d5cc 100644
--- a/arch/powerpc/platforms/powernv/vas-fault.c
+++ b/arch/powerpc/platforms/powernv/vas-fault.c
@@ -11,6 +11,7 @@
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/kthread.h>
+#include <linux/mmu_context.h>
#include <asm/icswx.h>
#include "vas.h"
@@ -25,6 +26,95 @@
#define VAS_FAULT_WIN_FIFO_SIZE (4 << 20)
/*
+ * Process valid CRBs in fault FIFO.
+ */
+irqreturn_t vas_fault_thread_fn(int irq, void *data)
+{
+ struct vas_instance *vinst = data;
+ struct coprocessor_request_block *crb, *entry;
+ struct coprocessor_request_block buf;
+ struct vas_window *window;
+ unsigned long flags;
+ void *fifo;
+
+ crb = &buf;
+
+ /*
+ * VAS can interrupt with multiple page faults. So process all
+ * valid CRBs within fault FIFO until reaches invalid CRB.
+ * NX updates nx_fault_stamp in CRB and pastes in fault FIFO.
+ * kernel retrives send window from parition send window ID
+ * (pswid) in nx_fault_stamp. So pswid should be valid and
+ * ccw[0] (in be) should be zero since this bit is reserved.
+ * If user space touches this bit, NX returns with "CRB format
+ * error".
+ *
+ * After reading CRB entry, invalidate it with pswid (set
+ * 0xffffffff) and ccw[0] (set to 1).
+ *
+ * In case kernel receives another interrupt with different page
+ * fault, CRBs are already processed by the previous handling. So
+ * will be returned from this function when it sees invalid CRB.
+ */
+ do {
+ mutex_lock(&vinst->mutex);
+
+ spin_lock_irqsave(&vinst->fault_lock, flags);
+ /*
+ * Advance the fault fifo pointer to next CRB.
+ * Use CRB_SIZE rather than sizeof(*crb) since the latter is
+ * aligned to CRB_ALIGN (256) but the CRB written to by VAS is
+ * only CRB_SIZE in len.
+ */
+ fifo = vinst->fault_fifo + (vinst->fault_crbs * CRB_SIZE);
+ entry = fifo;
+
+ if ((entry->stamp.nx.pswid == cpu_to_be32(FIFO_INVALID_ENTRY))
+ || (entry->ccw & cpu_to_be32(CCW0_INVALID))) {
+ atomic_set(&vinst->faults_in_progress, 0);
+ spin_unlock_irqrestore(&vinst->fault_lock, flags);
+ mutex_unlock(&vinst->mutex);
+ return IRQ_HANDLED;
+ }
+
+ spin_unlock_irqrestore(&vinst->fault_lock, flags);
+ vinst->fault_crbs++;
+ if (vinst->fault_crbs == (vinst->fault_fifo_size / CRB_SIZE))
+ vinst->fault_crbs = 0;
+
+ memcpy(crb, fifo, CRB_SIZE);
+ entry->stamp.nx.pswid = cpu_to_be32(FIFO_INVALID_ENTRY);
+ entry->ccw |= cpu_to_be32(CCW0_INVALID);
+ mutex_unlock(&vinst->mutex);
+
+ pr_devel("VAS[%d] fault_fifo %p, fifo %p, fault_crbs %d\n",
+ vinst->vas_id, vinst->fault_fifo, fifo,
+ vinst->fault_crbs);
+
+ window = vas_pswid_to_window(vinst,
+ be32_to_cpu(crb->stamp.nx.pswid));
+
+ if (IS_ERR(window)) {
+ /*
+ * We got an interrupt about a specific send
+ * window but we can't find that window and we can't
+ * even clean it up (return credit).
+ * But we should not get here.
+ */
+ pr_err("VAS[%d] fault_fifo %p, fifo %p, pswid 0x%x, fault_crbs %d bad CRB?\n",
+ vinst->vas_id, vinst->fault_fifo, fifo,
+ be32_to_cpu(crb->stamp.nx.pswid),
+ vinst->fault_crbs);
+
+ WARN_ON_ONCE(1);
+ atomic_set(&vinst->faults_in_progress, 0);
+ return IRQ_HANDLED;
+ }
+
+ } while (true);
+}
+
+/*
* Fault window is opened per VAS instance. NX pastes fault CRB in fault
* FIFO upon page faults.
*/
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index 1783fa9..1f31c18 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -1040,6 +1040,15 @@ struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop,
}
} else {
/*
+ * Interrupt hanlder or fault window setup failed. Means
+ * NX can not generate fault for page fault. So not
+ * opening for user space tx window.
+ */
+ if (!vinst->virq) {
+ rc = -ENODEV;
+ goto free_window;
+ }
+ /*
* A user mapping must ensure that context switch issues
* CP_ABORT for this thread.
*/
@@ -1254,3 +1263,54 @@ int vas_win_close(struct vas_window *window)
return 0;
}
EXPORT_SYMBOL_GPL(vas_win_close);
+
+struct vas_window *vas_pswid_to_window(struct vas_instance *vinst,
+ uint32_t pswid)
+{
+ struct vas_window *window;
+ int winid;
+
+ if (!pswid) {
+ pr_devel("%s: called for pswid 0!\n", __func__);
+ return ERR_PTR(-ESRCH);
+ }
+
+ decode_pswid(pswid, NULL, &winid);
+
+ if (winid >= VAS_WINDOWS_PER_CHIP)
+ return ERR_PTR(-ESRCH);
+
+ /*
+ * If application closes the window before the hardware
+ * returns the fault CRB, we should wait in vas_win_close()
+ * for the pending requests. so the window must be active
+ * and the process alive.
+ *
+ * If its a kernel process, we should not get any faults and
+ * should not get here.
+ */
+ window = vinst->windows[winid];
+
+ if (!window) {
+ pr_err("PSWID decode: Could not find window for winid %d pswid %d vinst 0x%p\n",
+ winid, pswid, vinst);
+ return NULL;
+ }
+
+ /*
+ * Do some sanity checks on the decoded window. Window should be
+ * NX GZIP user send window. FTW windows should not incur faults
+ * since their CRBs are ignored (not queued on FIFO or processed
+ * by NX).
+ */
+ if (!window->tx_win || !window->user_win || !window->nx_win ||
+ window->cop == VAS_COP_TYPE_FAULT ||
+ window->cop == VAS_COP_TYPE_FTW) {
+ pr_err("PSWID decode: id %d, tx %d, user %d, nx %d, cop %d\n",
+ winid, window->tx_win, window->user_win,
+ window->nx_win, window->cop);
+ WARN_ON(1);
+ }
+
+ return window;
+}
diff --git a/arch/powerpc/platforms/powernv/vas.c b/arch/powerpc/platforms/powernv/vas.c
index 557c8e4..3d9ba58 100644
--- a/arch/powerpc/platforms/powernv/vas.c
+++ b/arch/powerpc/platforms/powernv/vas.c
@@ -14,6 +14,8 @@
#include <linux/of_platform.h>
#include <linux/of_address.h>
#include <linux/of.h>
+#include <linux/irqdomain.h>
+#include <linux/interrupt.h>
#include <asm/prom.h>
#include <asm/xive.h>
@@ -24,9 +26,53 @@
static DEFINE_PER_CPU(int, cpu_vas_id);
+static irqreturn_t vas_fault_handler(int irq, void *dev_id)
+{
+ struct vas_instance *vinst = dev_id;
+ irqreturn_t ret = IRQ_WAKE_THREAD;
+ unsigned long flags;
+
+ /*
+ * NX can generate an interrupt for multiple faults. So the
+ * fault handler thread process all CRBs until finds invalid
+ * entry. In case if NX sees continuous faults, it is possible
+ * that the thread function entered with the first interrupt
+ * can execute and process all valid CRBs.
+ * So wake up thread only if the fault thread is not in progress.
+ */
+ spin_lock_irqsave(&vinst->fault_lock, flags);
+
+ if (atomic_read(&vinst->faults_in_progress))
+ ret = IRQ_HANDLED;
+ else
+ atomic_set(&vinst->faults_in_progress, 1);
+
+ spin_unlock_irqrestore(&vinst->fault_lock, flags);
+
+ return ret;
+}
+
static int vas_irq_fault_window_setup(struct vas_instance *vinst)
{
- return vas_setup_fault_window(vinst);
+ char devname[64];
+ int rc = 0;
+
+ snprintf(devname, sizeof(devname), "vas-%d", vinst->vas_id);
+ rc = request_threaded_irq(vinst->virq, vas_fault_handler,
+ vas_fault_thread_fn, 0, devname, vinst);
+
+ if (rc) {
+ pr_err("VAS[%d]: Request IRQ(%d) failed with %d\n",
+ vinst->vas_id, vinst->virq, rc);
+ goto out;
+ }
+
+ rc = vas_setup_fault_window(vinst);
+ if (rc)
+ free_irq(vinst->virq, vinst);
+
+out:
+ return rc;
}
static int init_vas_instance(struct platform_device *pdev)
@@ -109,6 +155,7 @@ static int init_vas_instance(struct platform_device *pdev)
list_add(&vinst->node, &vas_instances);
mutex_unlock(&vas_mutex);
+ spin_lock_init(&vinst->fault_lock);
/*
* IRQ and fault handling setup is needed only for user space
* send windows.
diff --git a/arch/powerpc/platforms/powernv/vas.h b/arch/powerpc/platforms/powernv/vas.h
index 6c4baf5..ecae7cd 100644
--- a/arch/powerpc/platforms/powernv/vas.h
+++ b/arch/powerpc/platforms/powernv/vas.h
@@ -326,7 +326,10 @@ struct vas_instance {
u64 irq_port;
int virq;
+ int fault_crbs;
int fault_fifo_size;
+ atomic_t faults_in_progress;
+ spinlock_t fault_lock;
void *fault_fifo;
struct vas_window *fault_win; /* Fault window */
@@ -424,6 +427,9 @@ struct vas_winctx {
extern void vas_window_init_dbgdir(struct vas_window *win);
extern void vas_window_free_dbgdir(struct vas_window *win);
extern int vas_setup_fault_window(struct vas_instance *vinst);
+extern irqreturn_t vas_fault_thread_fn(int irq, void *data);
+extern struct vas_window *vas_pswid_to_window(struct vas_instance *vinst,
+ uint32_t pswid);
static inline void vas_log_write(struct vas_window *win, char *name,
void *regptr, u64 val)
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 07/14] powerpc/vas: Register NX with fault window ID and IRQ port value
From: Haren Myneni @ 2020-03-19 6:16 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
For each user space send window, register NX with fault window ID
and port value so that NX paste CRBs in this fault FIFO when it
sees fault on the request buffer.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-window.c | 15 +++++++++++++--
arch/powerpc/platforms/powernv/vas.h | 15 +++++++++++++++
2 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index 1f31c18..acb6a22 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -373,7 +373,7 @@ int init_winctx_regs(struct vas_window *window, struct vas_winctx *winctx)
init_xlate_regs(window, winctx->user_win);
val = 0ULL;
- val = SET_FIELD(VAS_FAULT_TX_WIN, val, 0);
+ val = SET_FIELD(VAS_FAULT_TX_WIN, val, winctx->fault_win_id);
write_hvwc_reg(window, VREG(FAULT_TX_WIN), val);
/* In PowerNV, interrupts go to HV. */
@@ -748,6 +748,8 @@ static void init_winctx_for_rxwin(struct vas_window *rxwin,
winctx->min_scope = VAS_SCOPE_LOCAL;
winctx->max_scope = VAS_SCOPE_VECTORED_GROUP;
+ if (rxwin->vinst->virq)
+ winctx->irq_port = rxwin->vinst->irq_port;
}
static bool rx_win_args_valid(enum vas_cop_type cop,
@@ -944,13 +946,22 @@ static void init_winctx_for_txwin(struct vas_window *txwin,
winctx->lpid = txattr->lpid;
winctx->pidr = txattr->pidr;
winctx->rx_win_id = txwin->rxwin->winid;
+ /*
+ * IRQ and fault window setup is successful. Set fault window
+ * for the send window so that ready to handle faults.
+ */
+ if (txwin->vinst->virq)
+ winctx->fault_win_id = txwin->vinst->fault_win->winid;
winctx->dma_type = VAS_DMA_TYPE_INJECT;
winctx->tc_mode = txattr->tc_mode;
winctx->min_scope = VAS_SCOPE_LOCAL;
winctx->max_scope = VAS_SCOPE_VECTORED_GROUP;
+ if (txwin->vinst->virq)
+ winctx->irq_port = txwin->vinst->irq_port;
- winctx->pswid = 0;
+ winctx->pswid = txattr->pswid ? txattr->pswid :
+ encode_pswid(txwin->vinst->vas_id, txwin->winid);
}
static bool tx_win_args_valid(enum vas_cop_type cop,
diff --git a/arch/powerpc/platforms/powernv/vas.h b/arch/powerpc/platforms/powernv/vas.h
index ecae7cd..310b8a0 100644
--- a/arch/powerpc/platforms/powernv/vas.h
+++ b/arch/powerpc/platforms/powernv/vas.h
@@ -468,6 +468,21 @@ static inline u64 read_hvwc_reg(struct vas_window *win,
return in_be64(win->hvwc_map+reg);
}
+/*
+ * Encode/decode the Partition Send Window ID (PSWID) for a window in
+ * a way that we can uniquely identify any window in the system. i.e.
+ * we should be able to locate the 'struct vas_window' given the PSWID.
+ *
+ * Bits Usage
+ * 0:7 VAS id (8 bits)
+ * 8:15 Unused, 0 (3 bits)
+ * 16:31 Window id (16 bits)
+ */
+static inline u32 encode_pswid(int vasid, int winid)
+{
+ return ((u32)winid | (vasid << (31 - 7)));
+}
+
static inline void decode_pswid(u32 pswid, int *vasid, int *winid)
{
if (vasid)
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 08/14] powerpc/vas: Take reference to PID and mm for user space windows
From: Haren Myneni @ 2020-03-19 6:16 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
When process opens a window, its pid and tgid will be saved in vas_window
struct. This window will be closed when the process exits. Kernel handles
NX faults by updating CSB or send SEGV signal to pid if user space csb_addr
is invalid.
In multi-thread applications, a window can be opened by child thread, but
it will not be closed when this thread exits. Expects parent to clean up
all resources including NX windows. Child thread can send requests using
this window and can be killed before they are completed. But the pid
assigned to this thread can be reused for other task while requests are
pending. If the csb_addr passed in these requests is invalid, kernel will
end up sending signal to the wrong task.
To prevent reusing the pid, take references to pid and mm when the window
is opened and release them during window close.
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-debug.c | 2 +-
arch/powerpc/platforms/powernv/vas-window.c | 53 ++++++++++++++++++++++++++---
arch/powerpc/platforms/powernv/vas.h | 9 ++++-
3 files changed, 57 insertions(+), 7 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/vas-debug.c b/arch/powerpc/platforms/powernv/vas-debug.c
index 09e63df..ef9a717 100644
--- a/arch/powerpc/platforms/powernv/vas-debug.c
+++ b/arch/powerpc/platforms/powernv/vas-debug.c
@@ -38,7 +38,7 @@ static int info_show(struct seq_file *s, void *private)
seq_printf(s, "Type: %s, %s\n", cop_to_str(window->cop),
window->tx_win ? "Send" : "Receive");
- seq_printf(s, "Pid : %d\n", window->pid);
+ seq_printf(s, "Pid : %d\n", vas_window_pid(window));
unlock:
mutex_unlock(&vas_mutex);
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index acb6a22..e7641a5 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -12,6 +12,8 @@
#include <linux/log2.h>
#include <linux/rcupdate.h>
#include <linux/cred.h>
+#include <linux/sched/mm.h>
+#include <linux/mmu_context.h>
#include <asm/switch_to.h>
#include <asm/ppc-opcode.h>
#include "vas.h"
@@ -876,8 +878,6 @@ struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
rxwin->user_win = rxattr->user_win;
rxwin->cop = cop;
rxwin->wcreds_max = rxattr->wcreds_max ?: VAS_WCREDS_DEFAULT;
- if (rxattr->user_win)
- rxwin->pid = task_pid_vnr(current);
init_winctx_for_rxwin(rxwin, rxattr, &winctx);
init_winctx_regs(rxwin, &winctx);
@@ -1027,7 +1027,6 @@ struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop,
txwin->tx_win = 1;
txwin->rxwin = rxwin;
txwin->nx_win = txwin->rxwin->nx_win;
- txwin->pid = attr->pid;
txwin->user_win = attr->user_win;
txwin->wcreds_max = attr->wcreds_max ?: VAS_WCREDS_DEFAULT;
@@ -1068,8 +1067,43 @@ struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop,
goto free_window;
}
- set_vinst_win(vinst, txwin);
+ if (txwin->user_win) {
+ /*
+ * Window opened by child thread may not be closed when
+ * it exits. So take reference to its pid and release it
+ * when the window is free by parent thread.
+ * Acquire a reference to the task's pid to make sure
+ * pid will not be re-used - needed only for multithread
+ * applications.
+ */
+ txwin->pid = get_task_pid(current, PIDTYPE_PID);
+ /*
+ * Acquire a reference to the task's mm.
+ */
+ txwin->mm = get_task_mm(current);
+ if (!txwin->mm) {
+ put_pid(txwin->pid);
+ pr_err("VAS: pid(%d): mm_struct is not found\n",
+ current->pid);
+ rc = -EPERM;
+ goto free_window;
+ }
+
+ mmgrab(txwin->mm);
+ mmput(txwin->mm);
+ mm_context_add_copro(txwin->mm);
+ /*
+ * Process closes window during exit. In the case of
+ * multithread application, child can open window and
+ * can exit without closing it. Expects parent thread
+ * to use and close the window. So do not need to take
+ * pid reference for parent thread.
+ */
+ txwin->tgid = find_get_pid(task_tgid_vnr(current));
+ }
+
+ set_vinst_win(vinst, txwin);
return txwin;
free_window:
@@ -1266,8 +1300,17 @@ int vas_win_close(struct vas_window *window)
poll_window_castout(window);
/* if send window, drop reference to matching receive window */
- if (window->tx_win)
+ if (window->tx_win) {
+ if (window->user_win) {
+ /* Drop references to pid and mm */
+ put_pid(window->pid);
+ if (window->mm) {
+ mm_context_remove_copro(window->mm);
+ mmdrop(window->mm);
+ }
+ }
put_rx_win(window->rxwin);
+ }
vas_window_free(window);
diff --git a/arch/powerpc/platforms/powernv/vas.h b/arch/powerpc/platforms/powernv/vas.h
index 310b8a0..16aa8ec 100644
--- a/arch/powerpc/platforms/powernv/vas.h
+++ b/arch/powerpc/platforms/powernv/vas.h
@@ -353,7 +353,9 @@ struct vas_window {
bool user_win; /* True if user space window */
void *hvwc_map; /* HV window context */
void *uwc_map; /* OS/User window context */
- pid_t pid; /* Linux process id of owner */
+ struct pid *pid; /* Linux process id of owner */
+ struct pid *tgid; /* Thread group ID of owner */
+ struct mm_struct *mm; /* Linux process mm_struct */
int wcreds_max; /* Window credits */
char *dbgname;
@@ -431,6 +433,11 @@ struct vas_winctx {
extern struct vas_window *vas_pswid_to_window(struct vas_instance *vinst,
uint32_t pswid);
+static inline int vas_window_pid(struct vas_window *window)
+{
+ return pid_vnr(window->pid);
+}
+
static inline void vas_log_write(struct vas_window *win, char *name,
void *regptr, u64 val)
{
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 09/14] powerpc/vas: Update CSB and notify process for fault CRBs
From: Haren Myneni @ 2020-03-19 6:17 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
For each fault CRB, update fault address in CRB (fault_storage_addr)
and translation error status in CSB so that user space can touch the
fault address and resend the request. If the user space passed invalid
CSB address send signal to process with SIGSEGV.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-fault.c | 115 +++++++++++++++++++++++++++++
1 file changed, 115 insertions(+)
diff --git a/arch/powerpc/platforms/powernv/vas-fault.c b/arch/powerpc/platforms/powernv/vas-fault.c
index 1c6d5cc..6eceac5d 100644
--- a/arch/powerpc/platforms/powernv/vas-fault.c
+++ b/arch/powerpc/platforms/powernv/vas-fault.c
@@ -11,6 +11,7 @@
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/kthread.h>
+#include <linux/sched/signal.h>
#include <linux/mmu_context.h>
#include <asm/icswx.h>
@@ -26,6 +27,119 @@
#define VAS_FAULT_WIN_FIFO_SIZE (4 << 20)
/*
+ * Update the CSB to indicate a translation error.
+ *
+ * If we are unable to update the CSB means copy_to_user failed due to
+ * invalid csb_addr, send a signal to the process.
+ *
+ * Remaining settings in the CSB are based on wait_for_csb() of
+ * NX-GZIP.
+ */
+static void update_csb(struct vas_window *window,
+ struct coprocessor_request_block *crb)
+{
+ struct coprocessor_status_block csb;
+ struct kernel_siginfo info;
+ struct task_struct *tsk;
+ void __user *csb_addr;
+ struct pid *pid;
+ int rc;
+
+ /*
+ * NX user space windows can not be opened for task->mm=NULL
+ * and faults will not be generated for kernel requests.
+ */
+ if (WARN_ON_ONCE(!window->mm || !window->user_win))
+ return;
+
+ csb_addr = (void __user *)be64_to_cpu(crb->csb_addr);
+
+ memset(&csb, 0, sizeof(csb));
+ csb.cc = CSB_CC_TRANSLATION;
+ csb.ce = CSB_CE_TERMINATION;
+ csb.cs = 0;
+ csb.count = 0;
+
+ /*
+ * NX operates and returns in BE format as defined CRB struct.
+ * So return fault_storage_addr in BE as NX pastes in FIFO and
+ * expects user space to convert to CPU format.
+ */
+ csb.address = crb->stamp.nx.fault_storage_addr;
+ csb.flags = 0;
+
+ pid = window->pid;
+ tsk = get_pid_task(pid, PIDTYPE_PID);
+ /*
+ * Send window will be closed after processing all NX requests
+ * and process exits after closing all windows. In multi-thread
+ * applications, thread may not exists, but does not close FD
+ * (means send window) upon exit. Parent thread (tgid) can use
+ * and close the window later.
+ * pid and mm references are taken when window is opened by
+ * process (pid). So tgid is used only when child thread opens
+ * a window and exits without closing it in multithread tasks.
+ */
+ if (!tsk) {
+ pid = window->tgid;
+ tsk = get_pid_task(pid, PIDTYPE_PID);
+ /*
+ * Parent thread will be closing window during its exit.
+ * So should not get here.
+ */
+ if (WARN_ON_ONCE(!tsk))
+ return;
+ }
+
+ /* Return if the task is exiting. */
+ if (tsk->flags & PF_EXITING) {
+ put_task_struct(tsk);
+ return;
+ }
+
+ use_mm(window->mm);
+ rc = copy_to_user(csb_addr, &csb, sizeof(csb));
+ /*
+ * User space polls on csb.flags (first byte). So add barrier
+ * then copy first byte with csb flags update.
+ */
+ if (!rc) {
+ csb.flags = CSB_V;
+ smp_mb();
+ rc = copy_to_user(csb_addr, &csb, sizeof(u8));
+ }
+ unuse_mm(window->mm);
+ put_task_struct(tsk);
+
+ /* Success */
+ if (!rc)
+ return;
+
+ pr_debug("Invalid CSB address 0x%p signalling pid(%d)\n",
+ csb_addr, pid_vnr(pid));
+
+ clear_siginfo(&info);
+ info.si_signo = SIGSEGV;
+ info.si_errno = EFAULT;
+ info.si_code = SEGV_MAPERR;
+ info.si_addr = csb_addr;
+
+ /*
+ * process will be polling on csb.flags after request is sent to
+ * NX. So generally CSB update should not fail except when an
+ * application does not follow the process properly. So an error
+ * message will be displayed and leave it to user space whether
+ * to ignore or handle this signal.
+ */
+ rcu_read_lock();
+ rc = kill_pid_info(SIGSEGV, &info, pid);
+ rcu_read_unlock();
+
+ pr_devel("%s(): pid %d kill_proc_info() rc %d\n", __func__,
+ pid_vnr(pid), rc);
+}
+
+/*
* Process valid CRBs in fault FIFO.
*/
irqreturn_t vas_fault_thread_fn(int irq, void *data)
@@ -111,6 +225,7 @@ irqreturn_t vas_fault_thread_fn(int irq, void *data)
return IRQ_HANDLED;
}
+ update_csb(window, crb);
} while (true);
}
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 10/14] powerpc/vas: Print CRB and FIFO values
From: Haren Myneni @ 2020-03-19 6:18 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
Dump FIFO entries if could not find send window and print CRB
for debugging.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-fault.c | 41 ++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/arch/powerpc/platforms/powernv/vas-fault.c b/arch/powerpc/platforms/powernv/vas-fault.c
index 6eceac5d..40e1de4 100644
--- a/arch/powerpc/platforms/powernv/vas-fault.c
+++ b/arch/powerpc/platforms/powernv/vas-fault.c
@@ -26,6 +26,28 @@
*/
#define VAS_FAULT_WIN_FIFO_SIZE (4 << 20)
+static void dump_crb(struct coprocessor_request_block *crb)
+{
+ struct data_descriptor_entry *dde;
+ struct nx_fault_stamp *nx;
+
+ dde = &crb->source;
+ pr_devel("SrcDDE: addr 0x%llx, len %d, count %d, idx %d, flags %d\n",
+ be64_to_cpu(dde->address), be32_to_cpu(dde->length),
+ dde->count, dde->index, dde->flags);
+
+ dde = &crb->target;
+ pr_devel("TgtDDE: addr 0x%llx, len %d, count %d, idx %d, flags %d\n",
+ be64_to_cpu(dde->address), be32_to_cpu(dde->length),
+ dde->count, dde->index, dde->flags);
+
+ nx = &crb->stamp.nx;
+ pr_devel("NX Stamp: PSWID 0x%x, FSA 0x%llx, flags 0x%x, FS 0x%x\n",
+ be32_to_cpu(nx->pswid),
+ be64_to_cpu(crb->stamp.nx.fault_storage_addr),
+ nx->flags, nx->fault_status);
+}
+
/*
* Update the CSB to indicate a translation error.
*
@@ -139,6 +161,23 @@ static void update_csb(struct vas_window *window,
pid_vnr(pid), rc);
}
+static void dump_fifo(struct vas_instance *vinst, void *entry)
+{
+ unsigned long *end = vinst->fault_fifo + vinst->fault_fifo_size;
+ unsigned long *fifo = entry;
+ int i;
+
+ pr_err("Fault fifo size %d, Max crbs %d\n", vinst->fault_fifo_size,
+ vinst->fault_fifo_size / CRB_SIZE);
+
+ /* Dump 10 CRB entries or until end of FIFO */
+ pr_err("Fault FIFO Dump:\n");
+ for (i = 0; i < 10*(CRB_SIZE/8) && fifo < end; i += 4, fifo += 4) {
+ pr_err("[%.3d, %p]: 0x%.16lx 0x%.16lx 0x%.16lx 0x%.16lx\n",
+ i, fifo, *fifo, *(fifo+1), *(fifo+2), *(fifo+3));
+ }
+}
+
/*
* Process valid CRBs in fault FIFO.
*/
@@ -205,6 +244,7 @@ irqreturn_t vas_fault_thread_fn(int irq, void *data)
vinst->vas_id, vinst->fault_fifo, fifo,
vinst->fault_crbs);
+ dump_crb(crb);
window = vas_pswid_to_window(vinst,
be32_to_cpu(crb->stamp.nx.pswid));
@@ -215,6 +255,7 @@ irqreturn_t vas_fault_thread_fn(int irq, void *data)
* even clean it up (return credit).
* But we should not get here.
*/
+ dump_fifo(vinst, (void *)entry);
pr_err("VAS[%d] fault_fifo %p, fifo %p, pswid 0x%x, fault_crbs %d bad CRB?\n",
vinst->vas_id, vinst->fault_fifo, fifo,
be32_to_cpu(crb->stamp.nx.pswid),
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 11/14] powerpc/vas: Do not use default credits for receive window
From: Haren Myneni @ 2020-03-19 6:18 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
System checkstops if RxFIFO overruns with more requests than the
maximum possible number of CRBs allowed in FIFO at any time. So
max credits value (rxattr.wcreds_max) is set and is passed to
vas_rx_win_open() by the the driver.
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-window.c | 4 ++--
arch/powerpc/platforms/powernv/vas.h | 2 --
2 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index e7641a5..6658ccc 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -772,7 +772,7 @@ static bool rx_win_args_valid(enum vas_cop_type cop,
if (attr->rx_fifo_size > VAS_RX_FIFO_SIZE_MAX)
return false;
- if (attr->wcreds_max > VAS_RX_WCREDS_MAX)
+ if (!attr->wcreds_max)
return false;
if (attr->nx_win) {
@@ -877,7 +877,7 @@ struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
rxwin->nx_win = rxattr->nx_win;
rxwin->user_win = rxattr->user_win;
rxwin->cop = cop;
- rxwin->wcreds_max = rxattr->wcreds_max ?: VAS_WCREDS_DEFAULT;
+ rxwin->wcreds_max = rxattr->wcreds_max;
init_winctx_for_rxwin(rxwin, rxattr, &winctx);
init_winctx_regs(rxwin, &winctx);
diff --git a/arch/powerpc/platforms/powernv/vas.h b/arch/powerpc/platforms/powernv/vas.h
index 16aa8ec..bc728d7 100644
--- a/arch/powerpc/platforms/powernv/vas.h
+++ b/arch/powerpc/platforms/powernv/vas.h
@@ -101,11 +101,9 @@
/*
* Initial per-process credits.
* Max send window credits: 4K-1 (12-bits in VAS_TX_WCRED)
- * Max receive window credits: 64K-1 (16 bits in VAS_LRX_WCRED)
*
* TODO: Needs tuning for per-process credits
*/
-#define VAS_RX_WCREDS_MAX ((64 << 10) - 1)
#define VAS_TX_WCREDS_MAX ((4 << 10) - 1)
#define VAS_WCREDS_DEFAULT (1 << 10)
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 12/14] powerpc/vas: Return credits after handling fault
From: Haren Myneni @ 2020-03-19 6:19 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
NX expects OS to return credit for send window after processing each
fault. Also credit has to be returned even for fault window.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-fault.c | 9 +++++++++
arch/powerpc/platforms/powernv/vas-window.c | 17 +++++++++++++++++
arch/powerpc/platforms/powernv/vas.h | 1 +
3 files changed, 27 insertions(+)
diff --git a/arch/powerpc/platforms/powernv/vas-fault.c b/arch/powerpc/platforms/powernv/vas-fault.c
index 40e1de4..292f7ba 100644
--- a/arch/powerpc/platforms/powernv/vas-fault.c
+++ b/arch/powerpc/platforms/powernv/vas-fault.c
@@ -238,6 +238,10 @@ irqreturn_t vas_fault_thread_fn(int irq, void *data)
memcpy(crb, fifo, CRB_SIZE);
entry->stamp.nx.pswid = cpu_to_be32(FIFO_INVALID_ENTRY);
entry->ccw |= cpu_to_be32(CCW0_INVALID);
+ /*
+ * Return credit for the fault window.
+ */
+ vas_return_credit(vinst->fault_win, 0);
mutex_unlock(&vinst->mutex);
pr_devel("VAS[%d] fault_fifo %p, fifo %p, fault_crbs %d\n",
@@ -267,6 +271,11 @@ irqreturn_t vas_fault_thread_fn(int irq, void *data)
}
update_csb(window, crb);
+ /*
+ * Return credit for send window after processing
+ * fault CRB.
+ */
+ vas_return_credit(window, 1);
} while (true);
}
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index 6658ccc..20a2a8b 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -1318,6 +1318,23 @@ int vas_win_close(struct vas_window *window)
}
EXPORT_SYMBOL_GPL(vas_win_close);
+/*
+ * Return credit for the given window.
+ */
+void vas_return_credit(struct vas_window *window, bool tx)
+{
+ uint64_t val;
+
+ val = 0ULL;
+ if (tx) { /* send window */
+ val = SET_FIELD(VAS_TX_WCRED, val, 1);
+ write_hvwc_reg(window, VREG(TX_WCRED_ADDER), val);
+ } else {
+ val = SET_FIELD(VAS_LRX_WCRED, val, 1);
+ write_hvwc_reg(window, VREG(LRX_WCRED_ADDER), val);
+ }
+}
+
struct vas_window *vas_pswid_to_window(struct vas_instance *vinst,
uint32_t pswid)
{
diff --git a/arch/powerpc/platforms/powernv/vas.h b/arch/powerpc/platforms/powernv/vas.h
index bc728d7..8c39a7d 100644
--- a/arch/powerpc/platforms/powernv/vas.h
+++ b/arch/powerpc/platforms/powernv/vas.h
@@ -428,6 +428,7 @@ struct vas_winctx {
extern void vas_window_free_dbgdir(struct vas_window *win);
extern int vas_setup_fault_window(struct vas_instance *vinst);
extern irqreturn_t vas_fault_thread_fn(int irq, void *data);
+extern void vas_return_credit(struct vas_window *window, bool tx);
extern struct vas_window *vas_pswid_to_window(struct vas_instance *vinst,
uint32_t pswid);
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 13/14] powerpc/vas: Display process stuck message
From: Haren Myneni @ 2020-03-19 6:19 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
Process can not close send window until all requests are processed.
Means wait until window state is not busy and send credits are
returned. Display debug messages in case taking longer to close the
window.
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-window.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index 20a2a8b..d5754a8 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -1182,6 +1182,7 @@ static void poll_window_credits(struct vas_window *window)
{
u64 val;
int creds, mode;
+ int count = 0;
val = read_hvwc_reg(window, VREG(WINCTL));
if (window->tx_win)
@@ -1200,10 +1201,27 @@ static void poll_window_credits(struct vas_window *window)
creds = GET_FIELD(VAS_LRX_WCRED, val);
}
+ /*
+ * Takes around few milliseconds to complete all pending requests
+ * and return credits.
+ * TODO: Scan fault FIFO and invalidate CRBs points to this window
+ * and issue CRB Kill to stop all pending requests. Need only
+ * if there is a bug in NX or fault handling in kernel.
+ */
if (creds < window->wcreds_max) {
val = 0;
set_current_state(TASK_UNINTERRUPTIBLE);
schedule_timeout(msecs_to_jiffies(10));
+ count++;
+ /*
+ * Process can not close send window until all credits are
+ * returned.
+ */
+ if (!(count % 10000))
+ pr_debug("VAS: pid %d stuck. Waiting for credits returned for Window(%d). creds %d, Retries %d\n",
+ vas_window_pid(window), window->winid,
+ creds, count);
+
goto retry;
}
}
@@ -1217,6 +1235,7 @@ static void poll_window_busy_state(struct vas_window *window)
{
int busy;
u64 val;
+ int count = 0;
retry:
val = read_hvwc_reg(window, VREG(WIN_STATUS));
@@ -1225,6 +1244,15 @@ static void poll_window_busy_state(struct vas_window *window)
val = 0;
set_current_state(TASK_UNINTERRUPTIBLE);
schedule_timeout(msecs_to_jiffies(5));
+ count++;
+ /*
+ * Takes around few milliseconds to process all pending
+ * requests.
+ */
+ if (!(count % 10000))
+ pr_debug("VAS: pid %d stuck. Window (ID=%d) is in busy state. Retries %d\n",
+ vas_window_pid(window), window->winid, count);
+
goto retry;
}
}
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 14/14] powerpc/vas: Free send window in VAS instance after credits returned
From: Haren Myneni @ 2020-03-19 6:20 UTC (permalink / raw)
To: mpe; +Cc: mikey, herbert, npiggin, hch, oohall, sukadev, linuxppc-dev, ajd
In-Reply-To: <1584598120.9256.15237.camel@hbabu-laptop>
NX may be processing requests while trying to close window. Wait until
all credits are returned and then free send window from VAS instance.
Signed-off-by: Haren Myneni <haren@linux.ibm.com>
---
arch/powerpc/platforms/powernv/vas-window.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index d5754a8..a1d3fb1 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -1317,14 +1317,14 @@ int vas_win_close(struct vas_window *window)
unmap_paste_region(window);
- clear_vinst_win(window);
-
poll_window_busy_state(window);
unpin_close_window(window);
poll_window_credits(window);
+ clear_vinst_win(window);
+
poll_window_castout(window);
/* if send window, drop reference to matching receive window */
--
1.8.3.1
^ permalink raw reply related
* Re: [PATCH 1/4] hugetlbfs: add arch_hugetlb_valid_size
From: Christophe Leroy @ 2020-03-19 7:00 UTC (permalink / raw)
To: Mike Kravetz, linux-mm, linux-kernel, linux-arm-kernel,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, linux-doc
Cc: Albert Ou, Vasily Gorbik, Jonathan Corbet, Catalin Marinas,
Dave Hansen, Heiko Carstens, David S . Miller,
Christian Borntraeger, Ingo Molnar, Palmer Dabbelt, Paul Walmsley,
Paul Mackerras, Andrew Morton, Longpeng, Will Deacon,
Thomas Gleixner
In-Reply-To: <20200318220634.32100-2-mike.kravetz@oracle.com>
Le 18/03/2020 à 23:06, Mike Kravetz a écrit :
> The architecture independent routine hugetlb_default_setup sets up
> the default huge pages size. It has no way to verify if the passed
> value is valid, so it accepts it and attempts to validate at a later
> time. This requires undocumented cooperation between the arch specific
> and arch independent code.
>
> For architectures that support more than one huge page size, provide
> a routine arch_hugetlb_valid_size to validate a huge page size.
> hugetlb_default_setup can use this to validate passed values.
>
> arch_hugetlb_valid_size will also be used in a subsequent patch to
> move processing of the "hugepagesz=" in arch specific code to a common
> routine in arch independent code.
>
> Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
> ---
> arch/arm64/include/asm/hugetlb.h | 2 ++
> arch/arm64/mm/hugetlbpage.c | 19 ++++++++++++++-----
> arch/powerpc/include/asm/hugetlb.h | 3 +++
> arch/powerpc/mm/hugetlbpage.c | 20 +++++++++++++-------
> arch/riscv/include/asm/hugetlb.h | 3 +++
> arch/riscv/mm/hugetlbpage.c | 28 ++++++++++++++++++----------
> arch/s390/include/asm/hugetlb.h | 3 +++
> arch/s390/mm/hugetlbpage.c | 18 +++++++++++++-----
> arch/sparc/include/asm/hugetlb.h | 3 +++
> arch/sparc/mm/init_64.c | 23 ++++++++++++++++-------
> arch/x86/include/asm/hugetlb.h | 3 +++
> arch/x86/mm/hugetlbpage.c | 21 +++++++++++++++------
> include/linux/hugetlb.h | 7 +++++++
> mm/hugetlb.c | 16 +++++++++++++---
> 14 files changed, 126 insertions(+), 43 deletions(-)
>
[snip]
> diff --git a/arch/powerpc/include/asm/hugetlb.h b/arch/powerpc/include/asm/hugetlb.h
> index bd6504c28c2f..3b5939016955 100644
> --- a/arch/powerpc/include/asm/hugetlb.h
> +++ b/arch/powerpc/include/asm/hugetlb.h
> @@ -64,6 +64,9 @@ static inline void arch_clear_hugepage_flags(struct page *page)
> {
> }
>
> +#define arch_hugetlb_valid_size arch_hugetlb_valid_size
> +extern bool __init arch_hugetlb_valid_size(unsigned long long size);
Don't add 'extern' keyword, it is irrelevant for a function declaration.
checkpatch --strict doesn't like it either
(https://openpower.xyz/job/snowpatch/job/snowpatch-linux-checkpatch/12318//artifact/linux/checkpatch.log)
> +
> #include <asm-generic/hugetlb.h>
>
> #else /* ! CONFIG_HUGETLB_PAGE */
> diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
> index 33b3461d91e8..b78f660252f3 100644
> --- a/arch/powerpc/mm/hugetlbpage.c
> +++ b/arch/powerpc/mm/hugetlbpage.c
> @@ -558,7 +558,7 @@ unsigned long vma_mmu_pagesize(struct vm_area_struct *vma)
> return vma_kernel_pagesize(vma);
> }
>
> -static int __init add_huge_page_size(unsigned long long size)
> +bool __init arch_hugetlb_valid_size(unsigned long long size)
> {
> int shift = __ffs(size);
> int mmu_psize;
> @@ -566,20 +566,26 @@ static int __init add_huge_page_size(unsigned long long size)
> /* Check that it is a page size supported by the hardware and
> * that it fits within pagetable and slice limits. */
> if (size <= PAGE_SIZE || !is_power_of_2(size))
> - return -EINVAL;
> + return false;
>
> mmu_psize = check_and_get_huge_psize(shift);
> if (mmu_psize < 0)
> - return -EINVAL;
> + return false;
>
> BUG_ON(mmu_psize_defs[mmu_psize].shift != shift);
>
> - /* Return if huge page size has already been setup */
> - if (size_to_hstate(size))
> - return 0;
> + return true;
> +}
>
> - hugetlb_add_hstate(shift - PAGE_SHIFT);
> +static int __init add_huge_page_size(unsigned long long size)
> +{
> + int shift = __ffs(size);
> +
> + if (!arch_hugetlb_valid_size(size))
> + return -EINVAL;
>
> + if (!size_to_hstate(size))
> + hugetlb_add_hstate(shift - PAGE_SHIFT);
> return 0;
> }
>
[snip]
> diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c
> index 5bfd5aef5378..51e6208fdeec 100644
> --- a/arch/x86/mm/hugetlbpage.c
> +++ b/arch/x86/mm/hugetlbpage.c
> @@ -181,16 +181,25 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
> #endif /* CONFIG_HUGETLB_PAGE */
>
> #ifdef CONFIG_X86_64
> +bool __init arch_hugetlb_valid_size(unsigned long long size)
> +{
> + if (size == PMD_SIZE)
> + return true;
> + else if (size == PUD_SIZE && boot_cpu_has(X86_FEATURE_GBPAGES))
> + return true;
> + else
> + return false;
> +}
> +
> static __init int setup_hugepagesz(char *opt)
> {
> - unsigned long ps = memparse(opt, &opt);
> - if (ps == PMD_SIZE) {
> - hugetlb_add_hstate(PMD_SHIFT - PAGE_SHIFT);
> - } else if (ps == PUD_SIZE && boot_cpu_has(X86_FEATURE_GBPAGES)) {
> - hugetlb_add_hstate(PUD_SHIFT - PAGE_SHIFT);
> + unsigned long long ps = memparse(opt, &opt);
> +
> + if (arch_hugetlb_valid_size(ps)) {
> + hugetlb_add_hstate(ilog2(ps) - PAGE_SHIFT);
> } else {
> hugetlb_bad_size();
> - printk(KERN_ERR "hugepagesz: Unsupported page size %lu M\n",
> + printk(KERN_ERR "hugepagesz: Unsupported page size %llu M\n",
> ps >> 20);
Nowadays we use pr_err() instead of printk.
It would also likely allow you to have everything fit on a single line.
> return 0;
> }
> diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
> index b831e9fa1a26..33343eb980d0 100644
> --- a/include/linux/hugetlb.h
> +++ b/include/linux/hugetlb.h
> @@ -678,6 +678,13 @@ static inline spinlock_t *huge_pte_lockptr(struct hstate *h,
> return &mm->page_table_lock;
> }
>
> +#ifndef arch_hugetlb_valid_size
> +static inline bool arch_hugetlb_valid_size(unsigned long long size)
> +{
> + return (size == HPAGE_SIZE);
Not sure the ( ) are necessary.
> +}
> +#endif
> +
> #ifndef hugepages_supported
> /*
> * Some platform decide whether they support huge pages at boot
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index d8ebd876871d..2f99359b93af 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -3224,12 +3224,22 @@ static int __init hugetlb_nrpages_setup(char *s)
> }
> __setup("hugepages=", hugetlb_nrpages_setup);
>
> -static int __init hugetlb_default_setup(char *s)
> +static int __init default_hugepagesz_setup(char *s)
> {
> - default_hstate_size = memparse(s, &s);
> + unsigned long long size;
Why unsigned long long ?
default_hstate_size is long.
I can't imagine 32 bits platforms having a hugepage with a 64 bits size.
> + char *saved_s = s;
> +
> + size = memparse(s, &s);
The updated s is not reused after that so you can pass NULL instead of
&s and then you don't need the saved_s.
> +
> + if (!arch_hugetlb_valid_size(size)) {
> + pr_err("HugeTLB: unsupported default_hugepagesz %s\n", saved_s);
> + return 0;
> + }
> +
> + default_hstate_size = size;
> return 1;
> }
> -__setup("default_hugepagesz=", hugetlb_default_setup);
> +__setup("default_hugepagesz=", default_hugepagesz_setup);
>
> static unsigned int cpuset_mems_nr(unsigned int *array)
> {
>
Christophe
^ permalink raw reply
* [PATCH] Fix "[v3, 28/32] powerpc/64s: interrupt implement exit logic in C"
From: Nicholas Piggin @ 2020-03-19 6:59 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
Accessing thread_info flags can cause an SLB fault, so it must not
be done with MSR[RI]=0, which leads to SLB unrecoverable fault error.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/syscall_64.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/syscall_64.c
index 87d95b455b83..a56ae78f2d1d 100644
--- a/arch/powerpc/kernel/syscall_64.c
+++ b/arch/powerpc/kernel/syscall_64.c
@@ -313,12 +313,18 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
{
unsigned long *ti_flagsp = ¤t_thread_info()->flags;
unsigned long flags;
+ unsigned long ret = 0;
if (IS_ENABLED(CONFIG_PPC_BOOK3S) && unlikely(!(regs->msr & MSR_RI)))
unrecoverable_exception(regs);
BUG_ON(regs->msr & MSR_PR);
BUG_ON(!FULL_REGS(regs));
+ if (unlikely(*ti_flagsp & _TIF_EMULATE_STACK_STORE)) {
+ clear_bits(_TIF_EMULATE_STACK_STORE, ti_flagsp);
+ ret = 1;
+ }
+
local_irq_save(flags);
if (regs->softe == IRQS_ENABLED) {
@@ -370,10 +376,6 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
*/
kuap_restore_amr(regs);
- if (unlikely(*ti_flagsp & _TIF_EMULATE_STACK_STORE)) {
- clear_bits(_TIF_EMULATE_STACK_STORE, ti_flagsp);
- return 1;
- }
- return 0;
+ return ret;
}
#endif
--
2.23.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox