* Re: [PATCH v4 1/5] powerpc/mm/slice: Remove intermediate bitmap copy
From: Nicholas Piggin @ 2018-02-10 14:43 UTC (permalink / raw)
To: Christophe Leroy
Cc: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Scott Wood, aneesh.kumar, linux-kernel, linuxppc-dev
In-Reply-To: <01e8f783db8f4d4d41df91e0400a8634272b326f.1518226173.git.christophe.leroy@c-s.fr>
On Sat, 10 Feb 2018 13:54:25 +0100 (CET)
Christophe Leroy <christophe.leroy@c-s.fr> wrote:
> bitmap_or() and bitmap_andnot() can work properly with dst identical
> to src1 or src2. There is no need of an intermediate result bitmap
> that is copied back to dst in a second step.
Everyone seems to be working on the slice code all of a sudden. I
had the same change in my series I just posted, but I didn't notice
this one of yours earlier, and it's better split out, so this is
fine by me.
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
>
> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
> Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
> ---
> v2: New in v2
> v3: patch moved up front of the serie to avoid ephemeral slice_bitmap_copy() function in following patch
> v4: No change
>
> arch/powerpc/mm/slice.c | 12 ++++--------
> 1 file changed, 4 insertions(+), 8 deletions(-)
>
> diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
> index 23ec2c5e3b78..98b53d48968f 100644
> --- a/arch/powerpc/mm/slice.c
> +++ b/arch/powerpc/mm/slice.c
> @@ -388,21 +388,17 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
>
> static inline void slice_or_mask(struct slice_mask *dst, struct slice_mask *src)
> {
> - DECLARE_BITMAP(result, SLICE_NUM_HIGH);
> -
> dst->low_slices |= src->low_slices;
> - bitmap_or(result, dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
> - bitmap_copy(dst->high_slices, result, SLICE_NUM_HIGH);
> + bitmap_or(dst->high_slices, dst->high_slices, src->high_slices,
> + SLICE_NUM_HIGH);
> }
>
> static inline void slice_andnot_mask(struct slice_mask *dst, struct slice_mask *src)
> {
> - DECLARE_BITMAP(result, SLICE_NUM_HIGH);
> -
> dst->low_slices &= ~src->low_slices;
>
> - bitmap_andnot(result, dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
> - bitmap_copy(dst->high_slices, result, SLICE_NUM_HIGH);
> + bitmap_andnot(dst->high_slices, dst->high_slices, src->high_slices,
> + SLICE_NUM_HIGH);
> }
>
> #ifdef CONFIG_PPC_64K_PAGES
^ permalink raw reply
* Re: [PATCH v3 4/5] powerpc/mm: Allow up to 64 low slices
From: Christophe LEROY @ 2018-02-10 12:58 UTC (permalink / raw)
To: Aneesh Kumar K.V, Benjamin Herrenschmidt, Paul Mackerras,
Michael Ellerman, Scott Wood
Cc: linux-kernel, linuxppc-dev
In-Reply-To: <87po5t18ll.fsf@linux.vnet.ibm.com>
Le 29/01/2018 à 07:29, Aneesh Kumar K.V a écrit :
> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>
>> While the implementation of the "slices" address space allows
>> a significant amount of high slices, it limits the number of
>> low slices to 16 due to the use of a single u64 low_slices_psize
>> element in struct mm_context_t
>>
>> On the 8xx, the minimum slice size is the size of the area
>> covered by a single PMD entry, ie 4M in 4K pages mode and 64M in
>> 16K pages mode. This means we could have at least 64 slices.
>>
>> In order to override this limitation, this patch switches the
>> handling of low_slices_psize to char array as done already for
>> high_slices_psize. This allows to increase the number of low
>> slices to 64 on the 8xx.
>>
>
> Maybe update the subject to "make low slice also a bitmap".Also indicate
> that the bitmap functions optimize the operation if the bitmapsize les
> <= long ?
As explained below above the ---, this new version don't do that
anymore, as 64 slices is enough.
>
> Also switch the 8xx to higher value in the another patch?
Ok
Christophe
>
>> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
>> ---
>> v2: Usign slice_bitmap_xxx() macros instead of bitmap_xxx() functions.
>> v3: keep low_slices as a u64, this allows 64 slices which is enough.
>>
>> arch/powerpc/include/asm/book3s/64/mmu.h | 3 +-
>> arch/powerpc/include/asm/mmu-8xx.h | 7 +++-
>> arch/powerpc/include/asm/paca.h | 2 +-
>> arch/powerpc/include/asm/slice.h | 1 -
>> arch/powerpc/include/asm/slice_32.h | 2 ++
>> arch/powerpc/include/asm/slice_64.h | 2 ++
>> arch/powerpc/kernel/paca.c | 3 +-
>> arch/powerpc/mm/hash_utils_64.c | 13 ++++----
>> arch/powerpc/mm/slb_low.S | 8 +++--
>> arch/powerpc/mm/slice.c | 57 +++++++++++++++++---------------
>> 10 files changed, 56 insertions(+), 42 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
>> index c9448e19847a..b076a2d74c69 100644
>> --- a/arch/powerpc/include/asm/book3s/64/mmu.h
>> +++ b/arch/powerpc/include/asm/book3s/64/mmu.h
>> @@ -91,7 +91,8 @@ typedef struct {
>> struct npu_context *npu_context;
>>
>> #ifdef CONFIG_PPC_MM_SLICES
>> - u64 low_slices_psize; /* SLB page size encodings */
>> + /* SLB page size encodings*/
>> + unsigned char low_slices_psize[BITS_PER_LONG / BITS_PER_BYTE];
>> unsigned char high_slices_psize[SLICE_ARRAY_SIZE];
>> unsigned long slb_addr_limit;
>> #else
>> diff --git a/arch/powerpc/include/asm/mmu-8xx.h b/arch/powerpc/include/asm/mmu-8xx.h
>> index 5f89b6010453..5f37ba06b56c 100644
>> --- a/arch/powerpc/include/asm/mmu-8xx.h
>> +++ b/arch/powerpc/include/asm/mmu-8xx.h
>> @@ -164,6 +164,11 @@
>> */
>> #define SPRN_M_TW 799
>>
>> +#ifdef CONFIG_PPC_MM_SLICES
>> +#include <asm/slice_32.h>
>> +#define SLICE_ARRAY_SIZE (1 << (32 - SLICE_LOW_SHIFT - 1))
>> +#endif
>> +
>> #ifndef __ASSEMBLY__
>> typedef struct {
>> unsigned int id;
>> @@ -171,7 +176,7 @@ typedef struct {
>> unsigned long vdso_base;
>> #ifdef CONFIG_PPC_MM_SLICES
>> u16 user_psize; /* page size index */
>> - u64 low_slices_psize; /* page size encodings */
>> + unsigned char low_slices_psize[SLICE_ARRAY_SIZE];
>> unsigned char high_slices_psize[0];
>> unsigned long slb_addr_limit;
>> #endif
>> diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
>> index 23ac7fc0af23..a3e531fe9ac7 100644
>> --- a/arch/powerpc/include/asm/paca.h
>> +++ b/arch/powerpc/include/asm/paca.h
>> @@ -141,7 +141,7 @@ struct paca_struct {
>> #ifdef CONFIG_PPC_BOOK3S
>> mm_context_id_t mm_ctx_id;
>> #ifdef CONFIG_PPC_MM_SLICES
>> - u64 mm_ctx_low_slices_psize;
>> + unsigned char mm_ctx_low_slices_psize[BITS_PER_LONG / BITS_PER_BYTE];
>> unsigned char mm_ctx_high_slices_psize[SLICE_ARRAY_SIZE];
>> unsigned long mm_ctx_slb_addr_limit;
>> #else
>> diff --git a/arch/powerpc/include/asm/slice.h b/arch/powerpc/include/asm/slice.h
>> index 2b4b70de7e71..b67ba8faa507 100644
>> --- a/arch/powerpc/include/asm/slice.h
>> +++ b/arch/powerpc/include/asm/slice.h
>> @@ -16,7 +16,6 @@
>> #define HAVE_ARCH_UNMAPPED_AREA
>> #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN
>>
>> -#define SLICE_LOW_SHIFT 28
>> #define SLICE_LOW_TOP (0x100000000ull)
>> #define SLICE_NUM_LOW (SLICE_LOW_TOP >> SLICE_LOW_SHIFT)
>> #define GET_LOW_SLICE_INDEX(addr) ((addr) >> SLICE_LOW_SHIFT)
>> diff --git a/arch/powerpc/include/asm/slice_32.h b/arch/powerpc/include/asm/slice_32.h
>> index 7e27c0dfb913..349187c20100 100644
>> --- a/arch/powerpc/include/asm/slice_32.h
>> +++ b/arch/powerpc/include/asm/slice_32.h
>> @@ -2,6 +2,8 @@
>> #ifndef _ASM_POWERPC_SLICE_32_H
>> #define _ASM_POWERPC_SLICE_32_H
>>
>> +#define SLICE_LOW_SHIFT 26 /* 64 slices */
>> +
>> #define SLICE_HIGH_SHIFT 0
>> #define SLICE_NUM_HIGH 0ul
>> #define GET_HIGH_SLICE_INDEX(addr) (addr & 0)
>> diff --git a/arch/powerpc/include/asm/slice_64.h b/arch/powerpc/include/asm/slice_64.h
>> index 9d1c97b83010..0959475239c6 100644
>> --- a/arch/powerpc/include/asm/slice_64.h
>> +++ b/arch/powerpc/include/asm/slice_64.h
>> @@ -2,6 +2,8 @@
>> #ifndef _ASM_POWERPC_SLICE_64_H
>> #define _ASM_POWERPC_SLICE_64_H
>>
>> +#define SLICE_LOW_SHIFT 28
>> +
>
> You are moving the LOW_SHIFT here, may be you can fix that up in earlier
> patch as reviewed there.
Yes, done in earlier patch now.
>
>> #define SLICE_HIGH_SHIFT 40
>> #define SLICE_NUM_HIGH (H_PGTABLE_RANGE >> SLICE_HIGH_SHIFT)
>> #define GET_HIGH_SLICE_INDEX(addr) ((addr) >> SLICE_HIGH_SHIFT)
>> diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
>> index d6597038931d..8e1566bf82b8 100644
>> --- a/arch/powerpc/kernel/paca.c
>> +++ b/arch/powerpc/kernel/paca.c
>> @@ -264,7 +264,8 @@ void copy_mm_to_paca(struct mm_struct *mm)
>> #ifdef CONFIG_PPC_MM_SLICES
>> VM_BUG_ON(!mm->context.slb_addr_limit);
>> get_paca()->mm_ctx_slb_addr_limit = mm->context.slb_addr_limit;
>> - get_paca()->mm_ctx_low_slices_psize = context->low_slices_psize;
>> + memcpy(&get_paca()->mm_ctx_low_slices_psize,
>> + &context->low_slices_psize, sizeof(context->low_slices_psize));
>> memcpy(&get_paca()->mm_ctx_high_slices_psize,
>> &context->high_slices_psize, TASK_SLICE_ARRAY_SZ(mm));
>> #else /* CONFIG_PPC_MM_SLICES */
>> diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
>> index 655a5a9a183d..da696565b969 100644
>> --- a/arch/powerpc/mm/hash_utils_64.c
>> +++ b/arch/powerpc/mm/hash_utils_64.c
>> @@ -1097,19 +1097,18 @@ unsigned int hash_page_do_lazy_icache(unsigned int pp, pte_t pte, int trap)
>> #ifdef CONFIG_PPC_MM_SLICES
>> static unsigned int get_paca_psize(unsigned long addr)
>> {
>> - u64 lpsizes;
>> - unsigned char *hpsizes;
>> + unsigned char *psizes;
>> unsigned long index, mask_index;
>>
>> if (addr < SLICE_LOW_TOP) {
>> - lpsizes = get_paca()->mm_ctx_low_slices_psize;
>> + psizes = get_paca()->mm_ctx_low_slices_psize;
>> index = GET_LOW_SLICE_INDEX(addr);
>> - return (lpsizes >> (index * 4)) & 0xF;
>> + } else {
>> + psizes = get_paca()->mm_ctx_high_slices_psize;
>> + index = GET_HIGH_SLICE_INDEX(addr);
>> }
>> - hpsizes = get_paca()->mm_ctx_high_slices_psize;
>> - index = GET_HIGH_SLICE_INDEX(addr);
>> mask_index = index & 0x1;
>> - return (hpsizes[index >> 1] >> (mask_index * 4)) & 0xF;
>> + return (psizes[index >> 1] >> (mask_index * 4)) & 0xF;
>> }
>>
>> #else
>> diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S
>> index 2cf5ef3fc50d..2c7c717fd2ea 100644
>> --- a/arch/powerpc/mm/slb_low.S
>> +++ b/arch/powerpc/mm/slb_low.S
>> @@ -200,10 +200,12 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_1T_SEGMENT)
>> 5:
>> /*
>> * Handle lpsizes
>> - * r9 is get_paca()->context.low_slices_psize, r11 is index
>> + * r9 is get_paca()->context.low_slices_psize[index], r11 is mask_index
>> */
>> - ld r9,PACALOWSLICESPSIZE(r13)
>> - mr r11,r10
>> + srdi r11,r10,1 /* index */
>> + addi r9,r11,PACALOWSLICESPSIZE
>> + lbzx r9,r13,r9 /* r9 is lpsizes[r11] */
>> + rldicl r11,r10,0,63 /* r11 = r10 & 0x1 */
>> 6:
>> sldi r11,r11,2 /* index * 4 */
>> /* Extract the psize and multiply to get an array offset */
>> diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
>> index 549704dfa777..3d573a038d42 100644
>> --- a/arch/powerpc/mm/slice.c
>> +++ b/arch/powerpc/mm/slice.c
>> @@ -148,18 +148,20 @@ static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
>> static void slice_mask_for_size(struct mm_struct *mm, int psize, struct slice_mask *ret,
>> unsigned long high_limit)
>> {
>> - unsigned char *hpsizes;
>> + unsigned char *hpsizes, *lpsizes;
>> int index, mask_index;
>> unsigned long i;
>> - u64 lpsizes;
>>
>> ret->low_slices = 0;
>> slice_bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
>>
>> lpsizes = mm->context.low_slices_psize;
>> - for (i = 0; i < SLICE_NUM_LOW; i++)
>> - if (((lpsizes >> (i * 4)) & 0xf) == psize)
>> + for (i = 0; i < SLICE_NUM_LOW; i++) {
>> + mask_index = i & 0x1;
>> + index = i >> 1;
>> + if (((lpsizes[index] >> (mask_index * 4)) & 0xf) == psize)
>> ret->low_slices |= 1u << i;
>> + }
>>
>> if (high_limit <= SLICE_LOW_TOP)
>> return;
>> @@ -211,8 +213,7 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
>> {
>> int index, mask_index;
>> /* Write the new slice psize bits */
>> - unsigned char *hpsizes;
>> - u64 lpsizes;
>> + unsigned char *hpsizes, *lpsizes;
>> unsigned long i, flags;
>>
>> slice_dbg("slice_convert(mm=%p, psize=%d)\n", mm, psize);
>> @@ -225,12 +226,13 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
>>
>> lpsizes = mm->context.low_slices_psize;
>> for (i = 0; i < SLICE_NUM_LOW; i++)
>> - if (mask.low_slices & (1u << i))
>> - lpsizes = (lpsizes & ~(0xful << (i * 4))) |
>> - (((unsigned long)psize) << (i * 4));
>> -
>> - /* Assign the value back */
>> - mm->context.low_slices_psize = lpsizes;
>> + if (mask.low_slices & (1u << i)) {
>> + mask_index = i & 0x1;
>> + index = i >> 1;
>> + lpsizes[index] = (lpsizes[index] &
>> + ~(0xf << (mask_index * 4))) |
>> + (((unsigned long)psize) << (mask_index * 4));
>> + }
>>
>> hpsizes = mm->context.high_slices_psize;
>> for (i = 0; i < GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit); i++) {
>> @@ -629,7 +631,7 @@ unsigned long arch_get_unmapped_area_topdown(struct file *filp,
>>
>> unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr)
>> {
>> - unsigned char *hpsizes;
>> + unsigned char *psizes;
>> int index, mask_index;
>>
>> /*
>> @@ -643,15 +645,14 @@ unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr)
>> #endif
>> }
>> if (addr < SLICE_LOW_TOP) {
>> - u64 lpsizes;
>> - lpsizes = mm->context.low_slices_psize;
>> + psizes = mm->context.low_slices_psize;
>> index = GET_LOW_SLICE_INDEX(addr);
>> - return (lpsizes >> (index * 4)) & 0xf;
>> + } else {
>> + psizes = mm->context.high_slices_psize;
>> + index = GET_HIGH_SLICE_INDEX(addr);
>> }
>> - hpsizes = mm->context.high_slices_psize;
>> - index = GET_HIGH_SLICE_INDEX(addr);
>> mask_index = index & 0x1;
>> - return (hpsizes[index >> 1] >> (mask_index * 4)) & 0xf;
>> + return (psizes[index >> 1] >> (mask_index * 4)) & 0xf;
>> }
>> EXPORT_SYMBOL_GPL(get_slice_psize);
>>
>> @@ -672,8 +673,8 @@ EXPORT_SYMBOL_GPL(get_slice_psize);
>> void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
>> {
>> int index, mask_index;
>> - unsigned char *hpsizes;
>> - unsigned long flags, lpsizes;
>> + unsigned char *hpsizes, *lpsizes;
>> + unsigned long flags;
>> unsigned int old_psize;
>> int i;
>>
>> @@ -691,12 +692,14 @@ void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
>> wmb();
>>
>> lpsizes = mm->context.low_slices_psize;
>> - for (i = 0; i < SLICE_NUM_LOW; i++)
>> - if (((lpsizes >> (i * 4)) & 0xf) == old_psize)
>> - lpsizes = (lpsizes & ~(0xful << (i * 4))) |
>> - (((unsigned long)psize) << (i * 4));
>> - /* Assign the value back */
>> - mm->context.low_slices_psize = lpsizes;
>> + for (i = 0; i < SLICE_NUM_LOW; i++) {
>> + mask_index = i & 0x1;
>> + index = i >> 1;
>> + if (((lpsizes[index] >> (mask_index * 4)) & 0xf) == old_psize)
>> + lpsizes[index] = (lpsizes[index] &
>> + ~(0xf << (mask_index * 4))) |
>> + (((unsigned long)psize) << (mask_index * 4));
>> + }
>>
>> hpsizes = mm->context.high_slices_psize;
>> for (i = 0; i < SLICE_NUM_HIGH; i++) {
>> --
>> 2.13.3
^ permalink raw reply
* Re: [PATCH v3 2/5] powerpc/mm: Enhance 'slice' for supporting PPC32
From: Christophe LEROY @ 2018-02-10 12:55 UTC (permalink / raw)
To: Aneesh Kumar K.V, linuxppc-dev@lists.ozlabs.org
In-Reply-To: <87vafl18vu.fsf@linux.vnet.ibm.com>
Le 29/01/2018 à 07:23, Aneesh Kumar K.V a écrit :
> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>
>> In preparation for the following patch which will fix an issue on
>> the 8xx by re-using the 'slices', this patch enhances the
>> 'slices' implementation to support 32 bits CPUs.
>>
>> On PPC32, the address space is limited to 4Gbytes, hence only the low
>> slices will be used.
>>
>> This patch moves "slices" functions prototypes from page64.h to slice.h
>>
>> The high slices use bitmaps. As bitmap functions are not prepared to
>> handling bitmaps of size 0, the bitmap_xxx() calls are wrapped into
>> slice_bitmap_xxx() functions which will void on PPC32
>>
>> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
>> ---
>> v2: First patch of v1 serie split in two parts ; added slice_bitmap_xxx() macros.
>> v3: Moving slice related stuff in slice.h and slice_32/64.h
>> slice_bitmap_xxx() are now static inline functions and platform dependent
>> SLICE_LOW_TOP declared ull on PPC32 with correct casts allows to keep it 0x100000000
>>
>> arch/powerpc/include/asm/page.h | 1 +
>> arch/powerpc/include/asm/page_64.h | 59 ----------------------------------
>> arch/powerpc/include/asm/slice.h | 63 +++++++++++++++++++++++++++++++++++++
>> arch/powerpc/include/asm/slice_32.h | 56 +++++++++++++++++++++++++++++++++
>> arch/powerpc/include/asm/slice_64.h | 61 +++++++++++++++++++++++++++++++++++
>> arch/powerpc/mm/slice.c | 38 ++++++++++++----------
>> 6 files changed, 203 insertions(+), 75 deletions(-)
>> create mode 100644 arch/powerpc/include/asm/slice.h
>> create mode 100644 arch/powerpc/include/asm/slice_32.h
>> create mode 100644 arch/powerpc/include/asm/slice_64.h
>>
>> diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h
>> index 8da5d4c1cab2..d5f1c41b7dba 100644
>> --- a/arch/powerpc/include/asm/page.h
>> +++ b/arch/powerpc/include/asm/page.h
>> @@ -344,5 +344,6 @@ typedef struct page *pgtable_t;
>>
>> #include <asm-generic/memory_model.h>
>> #endif /* __ASSEMBLY__ */
>> +#include <asm/slice.h>
>>
>> #endif /* _ASM_POWERPC_PAGE_H */
>> diff --git a/arch/powerpc/include/asm/page_64.h b/arch/powerpc/include/asm/page_64.h
>> index 56234c6fcd61..af04acdb873f 100644
>> --- a/arch/powerpc/include/asm/page_64.h
>> +++ b/arch/powerpc/include/asm/page_64.h
>> @@ -86,65 +86,6 @@ extern u64 ppc64_pft_size;
>>
>> #endif /* __ASSEMBLY__ */
>>
>> -#ifdef CONFIG_PPC_MM_SLICES
>> -
>> -#define SLICE_LOW_SHIFT 28
>> -#define SLICE_HIGH_SHIFT 40
>> -
>> -#define SLICE_LOW_TOP (0x100000000ul)
>> -#define SLICE_NUM_LOW (SLICE_LOW_TOP >> SLICE_LOW_SHIFT)
>> -#define SLICE_NUM_HIGH (H_PGTABLE_RANGE >> SLICE_HIGH_SHIFT)
>> -
>> -#define GET_LOW_SLICE_INDEX(addr) ((addr) >> SLICE_LOW_SHIFT)
>> -#define GET_HIGH_SLICE_INDEX(addr) ((addr) >> SLICE_HIGH_SHIFT)
>> -
>> -#ifndef __ASSEMBLY__
>> -struct mm_struct;
>> -
>> -extern unsigned long slice_get_unmapped_area(unsigned long addr,
>> - unsigned long len,
>> - unsigned long flags,
>> - unsigned int psize,
>> - int topdown);
>> -
>> -extern unsigned int get_slice_psize(struct mm_struct *mm,
>> - unsigned long addr);
>> -
>> -extern void slice_set_user_psize(struct mm_struct *mm, unsigned int psize);
>> -extern void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
>> - unsigned long len, unsigned int psize);
>> -
>> -#endif /* __ASSEMBLY__ */
>> -#else
>> -#define slice_init()
>> -#ifdef CONFIG_PPC_BOOK3S_64
>> -#define get_slice_psize(mm, addr) ((mm)->context.user_psize)
>> -#define slice_set_user_psize(mm, psize) \
>> -do { \
>> - (mm)->context.user_psize = (psize); \
>> - (mm)->context.sllp = SLB_VSID_USER | mmu_psize_defs[(psize)].sllp; \
>> -} while (0)
>> -#else /* !CONFIG_PPC_BOOK3S_64 */
>> -#ifdef CONFIG_PPC_64K_PAGES
>> -#define get_slice_psize(mm, addr) MMU_PAGE_64K
>> -#else /* CONFIG_PPC_64K_PAGES */
>> -#define get_slice_psize(mm, addr) MMU_PAGE_4K
>> -#endif /* !CONFIG_PPC_64K_PAGES */
>> -#define slice_set_user_psize(mm, psize) do { BUG(); } while(0)
>> -#endif /* CONFIG_PPC_BOOK3S_64 */
>> -
>> -#define slice_set_range_psize(mm, start, len, psize) \
>> - slice_set_user_psize((mm), (psize))
>> -#endif /* CONFIG_PPC_MM_SLICES */
>> -
>> -#ifdef CONFIG_HUGETLB_PAGE
>> -
>> -#ifdef CONFIG_PPC_MM_SLICES
>> -#define HAVE_ARCH_HUGETLB_UNMAPPED_AREA
>> -#endif
>> -
>> -#endif /* !CONFIG_HUGETLB_PAGE */
>> -
>> #define VM_DATA_DEFAULT_FLAGS \
>> (is_32bit_task() ? \
>> VM_DATA_DEFAULT_FLAGS32 : VM_DATA_DEFAULT_FLAGS64)
>> diff --git a/arch/powerpc/include/asm/slice.h b/arch/powerpc/include/asm/slice.h
>> new file mode 100644
>> index 000000000000..2b4b70de7e71
>> --- /dev/null
>> +++ b/arch/powerpc/include/asm/slice.h
>> @@ -0,0 +1,63 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +#ifndef _ASM_POWERPC_SLICE_H
>> +#define _ASM_POWERPC_SLICE_H
>> +
>> +#ifdef CONFIG_PPC_MM_SLICES
>> +
>> +#ifdef CONFIG_PPC64
>> +#include <asm/slice_64.h>
>> +#else
>> +#include <asm/slice_32.h>
>> +#endif
>> +
>> +#ifdef CONFIG_HUGETLB_PAGE
>> +#define HAVE_ARCH_HUGETLB_UNMAPPED_AREA
>> +#endif
>> +#define HAVE_ARCH_UNMAPPED_AREA
>> +#define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN
>> +
>> +#define SLICE_LOW_SHIFT 28
>> +#define SLICE_LOW_TOP (0x100000000ull)
>> +#define SLICE_NUM_LOW (SLICE_LOW_TOP >> SLICE_LOW_SHIFT)
>> +#define GET_LOW_SLICE_INDEX(addr) ((addr) >> SLICE_LOW_SHIFT)
>> +
>> +#ifndef __ASSEMBLY__
>> +
>> +struct mm_struct;
>> +
>> +unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
>> + unsigned long flags, unsigned int psize,
>> + int topdown);
>> +
>> +unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr);
>> +
>> +void slice_set_user_psize(struct mm_struct *mm, unsigned int psize);
>> +void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
>> + unsigned long len, unsigned int psize);
>> +
>> +#else
>> +#define slice_init()
>> +#ifdef CONFIG_PPC_BOOK3S_64
>> +#define get_slice_psize(mm, addr) ((mm)->context.user_psize)
>> +#define slice_set_user_psize(mm, psize) \
>> +do { \
>> + (mm)->context.user_psize = (psize); \
>> + (mm)->context.sllp = SLB_VSID_USER | mmu_psize_defs[(psize)].sllp; \
>> +} while (0)
>> +#else /* !CONFIG_PPC_BOOK3S_64 */
>> +#ifdef CONFIG_PPC_64K_PAGES
>> +#define get_slice_psize(mm, addr) MMU_PAGE_64K
>> +#else /* CONFIG_PPC_64K_PAGES */
>> +#define get_slice_psize(mm, addr) MMU_PAGE_4K
>> +#endif /* !CONFIG_PPC_64K_PAGES */
>> +#define slice_set_user_psize(mm, psize) do { BUG(); } while(0)
>> +#endif /* CONFIG_PPC_BOOK3S_64 */
>> +
>
> That #ifder can really got into respective slice_32/64.h
Ok, created a nohash/64/slice.h for that.
>
>> +#define slice_set_range_psize(mm, start, len, psize) \
>> + slice_set_user_psize((mm), (psize))
>> +
>> +#endif /* __ASSEMBLY__ */
>> +
>> +#endif /* CONFIG_PPC_MM_SLICES */
>> +
>> +#endif /* _ASM_POWERPC_SLICE_H */
>> diff --git a/arch/powerpc/include/asm/slice_32.h b/arch/powerpc/include/asm/slice_32.h
>> new file mode 100644
>> index 000000000000..7e27c0dfb913
>> --- /dev/null
>> +++ b/arch/powerpc/include/asm/slice_32.h
>
> Can this be nohash/32/slice.h ? IIUC nobody else use slice
Ok
>
>
>> @@ -0,0 +1,56 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +#ifndef _ASM_POWERPC_SLICE_32_H
>> +#define _ASM_POWERPC_SLICE_32_H
>> +
>> +#define SLICE_HIGH_SHIFT 0
>> +#define SLICE_NUM_HIGH 0ul
>> +#define GET_HIGH_SLICE_INDEX(addr) (addr & 0)
>> +
>> +#ifndef __ASSEMBLY__
>> +
>> +static inline void slice_bitmap_zero(unsigned long *dst, unsigned int nbits)
>> +{
>> +}
>> +
>> +static inline int slice_bitmap_and(unsigned long *dst,
>> + const unsigned long *src1,
>> + const unsigned long *src2,
>> + unsigned int nbits)
>> +{
>> + return 0;
>> +}
>> +
>> +static inline void slice_bitmap_or(unsigned long *dst,
>> + const unsigned long *src1,
>> + const unsigned long *src2,
>> + unsigned int nbits)
>> +{
>> +}
>> +
>> +static inline int slice_bitmap_andnot(unsigned long *dst,
>> + const unsigned long *src1,
>> + const unsigned long *src2,
>> + unsigned int nbits)
>> +{
>> + return 0;
>> +}
>> +
>> +static inline int slice_bitmap_equal(const unsigned long *src1,
>> + const unsigned long *src2,
>> + unsigned int nbits)
>> +{
>> + return 1;
>> +}
>> +
>> +static inline int slice_bitmap_empty(const unsigned long *src, unsigned nbits)
>> +{
>> + return 1;
>> +}
>> +
>> +static inline void slice_bitmap_set(unsigned long *map, unsigned int start,
>> + unsigned int nbits)
>> +{
>> +}
>> +#endif /* __ASSEMBLY__ */
>> +
>> +#endif /* _ASM_POWERPC_SLICE_32_H */
>> diff --git a/arch/powerpc/include/asm/slice_64.h b/arch/powerpc/include/asm/slice_64.h
>> new file mode 100644
>> index 000000000000..9d1c97b83010
>> --- /dev/null
>> +++ b/arch/powerpc/include/asm/slice_64.h
>
> Can this be book3s/63/slice.h ?
Ok
>
>> @@ -0,0 +1,61 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +#ifndef _ASM_POWERPC_SLICE_64_H
>> +#define _ASM_POWERPC_SLICE_64_H
>> +
>> +#define SLICE_HIGH_SHIFT 40
>> +#define SLICE_NUM_HIGH (H_PGTABLE_RANGE >> SLICE_HIGH_SHIFT)
>> +#define GET_HIGH_SLICE_INDEX(addr) ((addr) >> SLICE_HIGH_SHIFT)
>> +
>
> Can you keep both LOW and HIGH details together. Even if 32 bit make a
> copy of that?
Ok
Christophe
>
>> +#ifndef __ASSEMBLY__
>> +
>> +#include <linux/bitmap.h>
>> +
>> +static inline void slice_bitmap_zero(unsigned long *dst, unsigned int nbits)
>> +{
>> + bitmap_zero(dst, nbits);
>> +}
>> +
>> +static inline int slice_bitmap_and(unsigned long *dst,
>> + const unsigned long *src1,
>> + const unsigned long *src2,
>> + unsigned int nbits)
>> +{
>> + return bitmap_and(dst, src1, src2, nbits);
>> +}
>> +
>> +static inline void slice_bitmap_or(unsigned long *dst,
>> + const unsigned long *src1,
>> + const unsigned long *src2,
>> + unsigned int nbits)
>> +{
>> + bitmap_or(dst, src1, src2, nbits);
>> +}
>> +
>> +static inline int slice_bitmap_andnot(unsigned long *dst,
>> + const unsigned long *src1,
>> + const unsigned long *src2,
>> + unsigned int nbits)
>> +{
>> + return bitmap_andnot(dst, src1, src2, nbits);
>> +}
>> +
>> +static inline int slice_bitmap_equal(const unsigned long *src1,
>> + const unsigned long *src2,
>> + unsigned int nbits)
>> +{
>> + return bitmap_equal(src1, src2, nbits);
>> +}
>> +
>> +static inline int slice_bitmap_empty(const unsigned long *src, unsigned nbits)
>> +{
>> + return bitmap_empty(src, nbits);
>> +}
>> +
>> +static inline void slice_bitmap_set(unsigned long *map, unsigned int start,
>> + unsigned int nbits)
>> +{
>> + bitmap_set(map, start, nbits);
>> +}
>> +#endif /* __ASSEMBLY__ */
>> +
>> +#endif /* _ASM_POWERPC_SLICE_64_H */
>> diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
>> index 98b53d48968f..549704dfa777 100644
>> --- a/arch/powerpc/mm/slice.c
>> +++ b/arch/powerpc/mm/slice.c
>> @@ -73,10 +73,11 @@ static void slice_range_to_mask(unsigned long start, unsigned long len,
>> unsigned long end = start + len - 1;
>>
>> ret->low_slices = 0;
>> - bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
>> + slice_bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
>>
>> if (start < SLICE_LOW_TOP) {
>> - unsigned long mend = min(end, (SLICE_LOW_TOP - 1));
>> + unsigned long mend = min(end,
>> + (unsigned long)(SLICE_LOW_TOP - 1));
>>
>> ret->low_slices = (1u << (GET_LOW_SLICE_INDEX(mend) + 1))
>> - (1u << GET_LOW_SLICE_INDEX(start));
>> @@ -87,7 +88,7 @@ static void slice_range_to_mask(unsigned long start, unsigned long len,
>> unsigned long align_end = ALIGN(end, (1UL << SLICE_HIGH_SHIFT));
>> unsigned long count = GET_HIGH_SLICE_INDEX(align_end) - start_index;
>>
>> - bitmap_set(ret->high_slices, start_index, count);
>> + slice_bitmap_set(ret->high_slices, start_index, count);
>> }
>> }
>>
>> @@ -113,11 +114,13 @@ static int slice_high_has_vma(struct mm_struct *mm, unsigned long slice)
>> unsigned long start = slice << SLICE_HIGH_SHIFT;
>> unsigned long end = start + (1ul << SLICE_HIGH_SHIFT);
>>
>> +#ifdef CONFIG_PPC64
>> /* Hack, so that each addresses is controlled by exactly one
>> * of the high or low area bitmaps, the first high area starts
>> * at 4GB, not 0 */
>> if (start == 0)
>> start = SLICE_LOW_TOP;
>> +#endif
>>
>> return !slice_area_is_free(mm, start, end - start);
>> }
>> @@ -128,7 +131,7 @@ static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
>> unsigned long i;
>>
>> ret->low_slices = 0;
>> - bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
>> + slice_bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
>>
>> for (i = 0; i < SLICE_NUM_LOW; i++)
>> if (!slice_low_has_vma(mm, i))
>> @@ -151,7 +154,7 @@ static void slice_mask_for_size(struct mm_struct *mm, int psize, struct slice_ma
>> u64 lpsizes;
>>
>> ret->low_slices = 0;
>> - bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
>> + slice_bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
>>
>> lpsizes = mm->context.low_slices_psize;
>> for (i = 0; i < SLICE_NUM_LOW; i++)
>> @@ -180,15 +183,16 @@ static int slice_check_fit(struct mm_struct *mm,
>> */
>> unsigned long slice_count = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
>>
>> - bitmap_and(result, mask.high_slices,
>> - available.high_slices, slice_count);
>> + slice_bitmap_and(result, mask.high_slices, available.high_slices,
>> + slice_count);
>>
>> return (mask.low_slices & available.low_slices) == mask.low_slices &&
>> - bitmap_equal(result, mask.high_slices, slice_count);
>> + slice_bitmap_equal(result, mask.high_slices, slice_count);
>> }
>>
>> static void slice_flush_segments(void *parm)
>> {
>> +#ifdef CONFIG_PPC64
>> struct mm_struct *mm = parm;
>> unsigned long flags;
>>
>> @@ -200,6 +204,7 @@ static void slice_flush_segments(void *parm)
>> local_irq_save(flags);
>> slb_flush_and_rebolt();
>> local_irq_restore(flags);
>> +#endif
>> }
>>
>> static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psize)
>> @@ -389,16 +394,16 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
>> static inline void slice_or_mask(struct slice_mask *dst, struct slice_mask *src)
>> {
>> dst->low_slices |= src->low_slices;
>> - bitmap_or(dst->high_slices, dst->high_slices, src->high_slices,
>> - SLICE_NUM_HIGH);
>> + slice_bitmap_or(dst->high_slices, dst->high_slices, src->high_slices,
>> + SLICE_NUM_HIGH);
>> }
>>
>> static inline void slice_andnot_mask(struct slice_mask *dst, struct slice_mask *src)
>> {
>> dst->low_slices &= ~src->low_slices;
>>
>> - bitmap_andnot(dst->high_slices, dst->high_slices, src->high_slices,
>> - SLICE_NUM_HIGH);
>> + slice_bitmap_andnot(dst->high_slices, dst->high_slices,
>> + src->high_slices, SLICE_NUM_HIGH);
>> }
>>
>> #ifdef CONFIG_PPC_64K_PAGES
>> @@ -446,14 +451,14 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
>> * init different masks
>> */
>> mask.low_slices = 0;
>> - bitmap_zero(mask.high_slices, SLICE_NUM_HIGH);
>> + slice_bitmap_zero(mask.high_slices, SLICE_NUM_HIGH);
>>
>> /* silence stupid warning */;
>> potential_mask.low_slices = 0;
>> - bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
>> + slice_bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
>>
>> compat_mask.low_slices = 0;
>> - bitmap_zero(compat_mask.high_slices, SLICE_NUM_HIGH);
>> + slice_bitmap_zero(compat_mask.high_slices, SLICE_NUM_HIGH);
>>
>> /* Sanity checks */
>> BUG_ON(mm->task_size == 0);
>> @@ -591,7 +596,8 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
>> convert:
>> slice_andnot_mask(&mask, &good_mask);
>> slice_andnot_mask(&mask, &compat_mask);
>> - if (mask.low_slices || !bitmap_empty(mask.high_slices, SLICE_NUM_HIGH)) {
>> + if (mask.low_slices ||
>> + !slice_bitmap_empty(mask.high_slices, SLICE_NUM_HIGH)) {
>> slice_convert(mm, mask, psize);
>> if (psize > MMU_PAGE_BASE)
>> on_each_cpu(slice_flush_segments, mm, 1);
>> --
>> 2.13.3
^ permalink raw reply
* [PATCH v4 4/5] powerpc/mm/slice: Allow up to 64 low slices
From: Christophe Leroy @ 2018-02-10 12:54 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Scott Wood, aneesh.kumar
Cc: linux-kernel, linuxppc-dev, Nicholas Piggin
In-Reply-To: <01e8f783db8f4d4d41df91e0400a8634272b326f.1518226173.git.christophe.leroy@c-s.fr>
While the implementation of the "slices" address space allows
a significant amount of high slices, it limits the number of
low slices to 16 due to the use of a single u64 low_slices_psize
element in struct mm_context_t
On the 8xx, the minimum slice size is the size of the area
covered by a single PMD entry, ie 4M in 4K pages mode and 64M in
16K pages mode. This means we could have at least 64 slices.
In order to override this limitation, this patch switches the
handling of low_slices_psize to char array as done already for
high_slices_psize.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v2: Using slice_bitmap_xxx() macros instead of bitmap_xxx() functions.
v3: keep low_slices as a u64, this allows 64 slices which is enough.
v4: Moved the 8xx specifics to next patch
arch/powerpc/include/asm/book3s/64/mmu.h | 3 +-
arch/powerpc/include/asm/mmu-8xx.h | 7 +++-
arch/powerpc/include/asm/paca.h | 2 +-
arch/powerpc/kernel/paca.c | 3 +-
arch/powerpc/mm/hash_utils_64.c | 13 ++++----
arch/powerpc/mm/slb_low.S | 8 +++--
arch/powerpc/mm/slice.c | 57 +++++++++++++++++---------------
7 files changed, 52 insertions(+), 41 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
index 0abeb0e2d616..bef6e39ed63a 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu.h
@@ -91,7 +91,8 @@ typedef struct {
struct npu_context *npu_context;
#ifdef CONFIG_PPC_MM_SLICES
- u64 low_slices_psize; /* SLB page size encodings */
+ /* SLB page size encodings*/
+ unsigned char low_slices_psize[BITS_PER_LONG / BITS_PER_BYTE];
unsigned char high_slices_psize[SLICE_ARRAY_SIZE];
unsigned long slb_addr_limit;
#else
diff --git a/arch/powerpc/include/asm/mmu-8xx.h b/arch/powerpc/include/asm/mmu-8xx.h
index b324ab46d838..d3d7e79140c6 100644
--- a/arch/powerpc/include/asm/mmu-8xx.h
+++ b/arch/powerpc/include/asm/mmu-8xx.h
@@ -186,6 +186,11 @@
#define M_APG2 0x00000040
#define M_APG3 0x00000060
+#ifdef CONFIG_PPC_MM_SLICES
+#include <asm/nohash/32/slice.h>
+#define SLICE_ARRAY_SIZE (1 << (32 - SLICE_LOW_SHIFT - 1))
+#endif
+
#ifndef __ASSEMBLY__
typedef struct {
unsigned int id;
@@ -193,7 +198,7 @@ typedef struct {
unsigned long vdso_base;
#ifdef CONFIG_PPC_MM_SLICES
u16 user_psize; /* page size index */
- u64 low_slices_psize; /* page size encodings */
+ unsigned char low_slices_psize[SLICE_ARRAY_SIZE];
unsigned char high_slices_psize[0];
unsigned long slb_addr_limit;
#endif
diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
index b62c31037cad..d2bf71dddbef 100644
--- a/arch/powerpc/include/asm/paca.h
+++ b/arch/powerpc/include/asm/paca.h
@@ -141,7 +141,7 @@ struct paca_struct {
#ifdef CONFIG_PPC_BOOK3S
mm_context_id_t mm_ctx_id;
#ifdef CONFIG_PPC_MM_SLICES
- u64 mm_ctx_low_slices_psize;
+ unsigned char mm_ctx_low_slices_psize[BITS_PER_LONG / BITS_PER_BYTE];
unsigned char mm_ctx_high_slices_psize[SLICE_ARRAY_SIZE];
unsigned long mm_ctx_slb_addr_limit;
#else
diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
index 95ffedf14885..2fd563d05831 100644
--- a/arch/powerpc/kernel/paca.c
+++ b/arch/powerpc/kernel/paca.c
@@ -265,7 +265,8 @@ void copy_mm_to_paca(struct mm_struct *mm)
#ifdef CONFIG_PPC_MM_SLICES
VM_BUG_ON(!mm->context.slb_addr_limit);
get_paca()->mm_ctx_slb_addr_limit = mm->context.slb_addr_limit;
- get_paca()->mm_ctx_low_slices_psize = context->low_slices_psize;
+ memcpy(&get_paca()->mm_ctx_low_slices_psize,
+ &context->low_slices_psize, sizeof(context->low_slices_psize));
memcpy(&get_paca()->mm_ctx_high_slices_psize,
&context->high_slices_psize, TASK_SLICE_ARRAY_SZ(mm));
#else /* CONFIG_PPC_MM_SLICES */
diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
index 7d07c7e17db6..2c1f4dac1098 100644
--- a/arch/powerpc/mm/hash_utils_64.c
+++ b/arch/powerpc/mm/hash_utils_64.c
@@ -1109,19 +1109,18 @@ unsigned int hash_page_do_lazy_icache(unsigned int pp, pte_t pte, int trap)
#ifdef CONFIG_PPC_MM_SLICES
static unsigned int get_paca_psize(unsigned long addr)
{
- u64 lpsizes;
- unsigned char *hpsizes;
+ unsigned char *psizes;
unsigned long index, mask_index;
if (addr < SLICE_LOW_TOP) {
- lpsizes = get_paca()->mm_ctx_low_slices_psize;
+ psizes = get_paca()->mm_ctx_low_slices_psize;
index = GET_LOW_SLICE_INDEX(addr);
- return (lpsizes >> (index * 4)) & 0xF;
+ } else {
+ psizes = get_paca()->mm_ctx_high_slices_psize;
+ index = GET_HIGH_SLICE_INDEX(addr);
}
- hpsizes = get_paca()->mm_ctx_high_slices_psize;
- index = GET_HIGH_SLICE_INDEX(addr);
mask_index = index & 0x1;
- return (hpsizes[index >> 1] >> (mask_index * 4)) & 0xF;
+ return (psizes[index >> 1] >> (mask_index * 4)) & 0xF;
}
#else
diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S
index 2cf5ef3fc50d..2c7c717fd2ea 100644
--- a/arch/powerpc/mm/slb_low.S
+++ b/arch/powerpc/mm/slb_low.S
@@ -200,10 +200,12 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_1T_SEGMENT)
5:
/*
* Handle lpsizes
- * r9 is get_paca()->context.low_slices_psize, r11 is index
+ * r9 is get_paca()->context.low_slices_psize[index], r11 is mask_index
*/
- ld r9,PACALOWSLICESPSIZE(r13)
- mr r11,r10
+ srdi r11,r10,1 /* index */
+ addi r9,r11,PACALOWSLICESPSIZE
+ lbzx r9,r13,r9 /* r9 is lpsizes[r11] */
+ rldicl r11,r10,0,63 /* r11 = r10 & 0x1 */
6:
sldi r11,r11,2 /* index * 4 */
/* Extract the psize and multiply to get an array offset */
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 549704dfa777..3d573a038d42 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -148,18 +148,20 @@ static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
static void slice_mask_for_size(struct mm_struct *mm, int psize, struct slice_mask *ret,
unsigned long high_limit)
{
- unsigned char *hpsizes;
+ unsigned char *hpsizes, *lpsizes;
int index, mask_index;
unsigned long i;
- u64 lpsizes;
ret->low_slices = 0;
slice_bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
lpsizes = mm->context.low_slices_psize;
- for (i = 0; i < SLICE_NUM_LOW; i++)
- if (((lpsizes >> (i * 4)) & 0xf) == psize)
+ for (i = 0; i < SLICE_NUM_LOW; i++) {
+ mask_index = i & 0x1;
+ index = i >> 1;
+ if (((lpsizes[index] >> (mask_index * 4)) & 0xf) == psize)
ret->low_slices |= 1u << i;
+ }
if (high_limit <= SLICE_LOW_TOP)
return;
@@ -211,8 +213,7 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
{
int index, mask_index;
/* Write the new slice psize bits */
- unsigned char *hpsizes;
- u64 lpsizes;
+ unsigned char *hpsizes, *lpsizes;
unsigned long i, flags;
slice_dbg("slice_convert(mm=%p, psize=%d)\n", mm, psize);
@@ -225,12 +226,13 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
lpsizes = mm->context.low_slices_psize;
for (i = 0; i < SLICE_NUM_LOW; i++)
- if (mask.low_slices & (1u << i))
- lpsizes = (lpsizes & ~(0xful << (i * 4))) |
- (((unsigned long)psize) << (i * 4));
-
- /* Assign the value back */
- mm->context.low_slices_psize = lpsizes;
+ if (mask.low_slices & (1u << i)) {
+ mask_index = i & 0x1;
+ index = i >> 1;
+ lpsizes[index] = (lpsizes[index] &
+ ~(0xf << (mask_index * 4))) |
+ (((unsigned long)psize) << (mask_index * 4));
+ }
hpsizes = mm->context.high_slices_psize;
for (i = 0; i < GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit); i++) {
@@ -629,7 +631,7 @@ unsigned long arch_get_unmapped_area_topdown(struct file *filp,
unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr)
{
- unsigned char *hpsizes;
+ unsigned char *psizes;
int index, mask_index;
/*
@@ -643,15 +645,14 @@ unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr)
#endif
}
if (addr < SLICE_LOW_TOP) {
- u64 lpsizes;
- lpsizes = mm->context.low_slices_psize;
+ psizes = mm->context.low_slices_psize;
index = GET_LOW_SLICE_INDEX(addr);
- return (lpsizes >> (index * 4)) & 0xf;
+ } else {
+ psizes = mm->context.high_slices_psize;
+ index = GET_HIGH_SLICE_INDEX(addr);
}
- hpsizes = mm->context.high_slices_psize;
- index = GET_HIGH_SLICE_INDEX(addr);
mask_index = index & 0x1;
- return (hpsizes[index >> 1] >> (mask_index * 4)) & 0xf;
+ return (psizes[index >> 1] >> (mask_index * 4)) & 0xf;
}
EXPORT_SYMBOL_GPL(get_slice_psize);
@@ -672,8 +673,8 @@ EXPORT_SYMBOL_GPL(get_slice_psize);
void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
{
int index, mask_index;
- unsigned char *hpsizes;
- unsigned long flags, lpsizes;
+ unsigned char *hpsizes, *lpsizes;
+ unsigned long flags;
unsigned int old_psize;
int i;
@@ -691,12 +692,14 @@ void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
wmb();
lpsizes = mm->context.low_slices_psize;
- for (i = 0; i < SLICE_NUM_LOW; i++)
- if (((lpsizes >> (i * 4)) & 0xf) == old_psize)
- lpsizes = (lpsizes & ~(0xful << (i * 4))) |
- (((unsigned long)psize) << (i * 4));
- /* Assign the value back */
- mm->context.low_slices_psize = lpsizes;
+ for (i = 0; i < SLICE_NUM_LOW; i++) {
+ mask_index = i & 0x1;
+ index = i >> 1;
+ if (((lpsizes[index] >> (mask_index * 4)) & 0xf) == old_psize)
+ lpsizes[index] = (lpsizes[index] &
+ ~(0xf << (mask_index * 4))) |
+ (((unsigned long)psize) << (mask_index * 4));
+ }
hpsizes = mm->context.high_slices_psize;
for (i = 0; i < SLICE_NUM_HIGH; i++) {
--
2.13.3
^ permalink raw reply related
* [PATCH v4 3/5] powerpc/mm/slice: Fix hugepage allocation at hint address on 8xx
From: Christophe Leroy @ 2018-02-10 12:54 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Scott Wood, aneesh.kumar
Cc: linux-kernel, linuxppc-dev, Nicholas Piggin
In-Reply-To: <01e8f783db8f4d4d41df91e0400a8634272b326f.1518226173.git.christophe.leroy@c-s.fr>
On the 8xx, the page size is set in the PMD entry and applies to
all pages of the page table pointed by the said PMD entry.
When an app has some regular pages allocated (e.g. see below) and tries
to mmap() a huge page at a hint address covered by the same PMD entry,
the kernel accepts the hint allthough the 8xx cannot handle different
page sizes in the same PMD entry.
10000000-10001000 r-xp 00000000 00:0f 2597 /root/malloc
10010000-10011000 rwxp 00000000 00:0f 2597 /root/malloc
mmap(0x10080000, 524288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|0x40000, -1, 0) = 0x10080000
This results the app remaining forever in do_page_fault()/hugetlb_fault()
and when interrupting that app, we get the following warning:
[162980.035629] WARNING: CPU: 0 PID: 2777 at arch/powerpc/mm/hugetlbpage.c:354 hugetlb_free_pgd_range+0xc8/0x1e4
[162980.035699] CPU: 0 PID: 2777 Comm: malloc Tainted: G W 4.14.6 #85
[162980.035744] task: c67e2c00 task.stack: c668e000
[162980.035783] NIP: c000fe18 LR: c00e1eec CTR: c00f90c0
[162980.035830] REGS: c668fc20 TRAP: 0700 Tainted: G W (4.14.6)
[162980.035854] MSR: 00029032 <EE,ME,IR,DR,RI> CR: 24044224 XER: 20000000
[162980.036003]
[162980.036003] GPR00: c00e1eec c668fcd0 c67e2c00 00000010 c6869410 10080000 00000000 77fb4000
[162980.036003] GPR08: ffff0001 0683c001 00000000 ffffff80 44028228 10018a34 00004008 418004fc
[162980.036003] GPR16: c668e000 00040100 c668e000 c06c0000 c668fe78 c668e000 c6835ba0 c668fd48
[162980.036003] GPR24: 00000000 73ffffff 74000000 00000001 77fb4000 100fffff 10100000 10100000
[162980.036743] NIP [c000fe18] hugetlb_free_pgd_range+0xc8/0x1e4
[162980.036839] LR [c00e1eec] free_pgtables+0x12c/0x150
[162980.036861] Call Trace:
[162980.036939] [c668fcd0] [c00f0774] unlink_anon_vmas+0x1c4/0x214 (unreliable)
[162980.037040] [c668fd10] [c00e1eec] free_pgtables+0x12c/0x150
[162980.037118] [c668fd40] [c00eabac] exit_mmap+0xe8/0x1b4
[162980.037210] [c668fda0] [c0019710] mmput.part.9+0x20/0xd8
[162980.037301] [c668fdb0] [c001ecb0] do_exit+0x1f0/0x93c
[162980.037386] [c668fe00] [c001f478] do_group_exit+0x40/0xcc
[162980.037479] [c668fe10] [c002a76c] get_signal+0x47c/0x614
[162980.037570] [c668fe70] [c0007840] do_signal+0x54/0x244
[162980.037654] [c668ff30] [c0007ae8] do_notify_resume+0x34/0x88
[162980.037744] [c668ff40] [c000dae8] do_user_signal+0x74/0xc4
[162980.037781] Instruction dump:
[162980.037821] 7fdff378 81370000 54a3463a 80890020 7d24182e 7c841a14 712a0004 4082ff94
[162980.038014] 2f890000 419e0010 712a0ff0 408200e0 <0fe00000> 54a9000a 7f984840 419d0094
[162980.038216] ---[ end trace c0ceeca8e7a5800a ]---
[162980.038754] BUG: non-zero nr_ptes on freeing mm: 1
[162985.363322] BUG: non-zero nr_ptes on freeing mm: -1
In order to fix this, this patch uses the address space "slices"
implemented for BOOK3S/64 and enhanced to support PPC32 by the
preceding patch.
This patch modifies the context.id on the 8xx to be in the range
[1:16] instead of [0:15] in order to identify context.id == 0 as
not initialised contexts as done on BOOK3S
This patch activates CONFIG_PPC_MM_SLICES when CONFIG_HUGETLB_PAGE is
selected for the 8xx
Alltough we could in theory have as many slices as PMD entries, the
current slices implementation limits the number of low slices to 16.
This limitation is not preventing us to fix the initial issue allthough
it is suboptimal. It will be cured in a subsequent patch.
Fixes: 4b91428699477 ("powerpc/8xx: Implement support of hugepages")
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
v2: First patch of v1 serie split in two parts
v3: No changes
v4: No changes
arch/powerpc/include/asm/mmu-8xx.h | 6 ++++++
arch/powerpc/kernel/setup-common.c | 2 ++
arch/powerpc/mm/8xx_mmu.c | 2 +-
arch/powerpc/mm/hugetlbpage.c | 2 ++
arch/powerpc/mm/mmu_context_nohash.c | 18 ++++++++++++++++--
arch/powerpc/platforms/Kconfig.cputype | 1 +
6 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/include/asm/mmu-8xx.h b/arch/powerpc/include/asm/mmu-8xx.h
index 2f806e329648..b324ab46d838 100644
--- a/arch/powerpc/include/asm/mmu-8xx.h
+++ b/arch/powerpc/include/asm/mmu-8xx.h
@@ -191,6 +191,12 @@ typedef struct {
unsigned int id;
unsigned int active;
unsigned long vdso_base;
+#ifdef CONFIG_PPC_MM_SLICES
+ u16 user_psize; /* page size index */
+ u64 low_slices_psize; /* page size encodings */
+ unsigned char high_slices_psize[0];
+ unsigned long slb_addr_limit;
+#endif
} mm_context_t;
#define PHYS_IMMR_BASE (mfspr(SPRN_IMMR) & 0xfff80000)
diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c
index d73ec518ef80..a6002f9449b1 100644
--- a/arch/powerpc/kernel/setup-common.c
+++ b/arch/powerpc/kernel/setup-common.c
@@ -919,6 +919,8 @@ void __init setup_arch(char **cmdline_p)
#ifdef CONFIG_PPC64
if (!radix_enabled())
init_mm.context.slb_addr_limit = DEFAULT_MAP_WINDOW_USER64;
+#elif defined(CONFIG_PPC_8xx)
+ init_mm.context.slb_addr_limit = DEFAULT_MAP_WINDOW;
#else
#error "context.addr_limit not initialized."
#endif
diff --git a/arch/powerpc/mm/8xx_mmu.c b/arch/powerpc/mm/8xx_mmu.c
index 849f50cd62f2..cf77d755246d 100644
--- a/arch/powerpc/mm/8xx_mmu.c
+++ b/arch/powerpc/mm/8xx_mmu.c
@@ -192,7 +192,7 @@ void set_context(unsigned long id, pgd_t *pgd)
mtspr(SPRN_M_TW, __pa(pgd) - offset);
/* Update context */
- mtspr(SPRN_M_CASID, id);
+ mtspr(SPRN_M_CASID, id - 1);
/* sync */
mb();
}
diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
index 876da2bc1796..590be3fa0ce2 100644
--- a/arch/powerpc/mm/hugetlbpage.c
+++ b/arch/powerpc/mm/hugetlbpage.c
@@ -553,9 +553,11 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
struct hstate *hstate = hstate_file(file);
int mmu_psize = shift_to_mmu_psize(huge_page_shift(hstate));
+#ifdef CONFIG_PPC_RADIX_MMU
if (radix_enabled())
return radix__hugetlb_get_unmapped_area(file, addr, len,
pgoff, flags);
+#endif
return slice_get_unmapped_area(addr, len, flags, mmu_psize, 1);
}
#endif
diff --git a/arch/powerpc/mm/mmu_context_nohash.c b/arch/powerpc/mm/mmu_context_nohash.c
index 4554d6527682..d98f7e5c141b 100644
--- a/arch/powerpc/mm/mmu_context_nohash.c
+++ b/arch/powerpc/mm/mmu_context_nohash.c
@@ -331,6 +331,20 @@ int init_new_context(struct task_struct *t, struct mm_struct *mm)
{
pr_hard("initing context for mm @%p\n", mm);
+#ifdef CONFIG_PPC_MM_SLICES
+ if (!mm->context.slb_addr_limit)
+ mm->context.slb_addr_limit = DEFAULT_MAP_WINDOW;
+
+ /*
+ * We have MMU_NO_CONTEXT set to be ~0. Hence check
+ * explicitly against context.id == 0. This ensures that we properly
+ * initialize context slice details for newly allocated mm's (which will
+ * have id == 0) and don't alter context slice inherited via fork (which
+ * will have id != 0).
+ */
+ if (mm->context.id == 0)
+ slice_set_user_psize(mm, mmu_virtual_psize);
+#endif
mm->context.id = MMU_NO_CONTEXT;
mm->context.active = 0;
return 0;
@@ -428,8 +442,8 @@ void __init mmu_context_init(void)
* -- BenH
*/
if (mmu_has_feature(MMU_FTR_TYPE_8xx)) {
- first_context = 0;
- last_context = 15;
+ first_context = 1;
+ last_context = 16;
no_selective_tlbil = true;
} else if (mmu_has_feature(MMU_FTR_TYPE_47x)) {
first_context = 1;
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index a429d859f15d..5a8b1bf1e819 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -326,6 +326,7 @@ config PPC_BOOK3E_MMU
config PPC_MM_SLICES
bool
default y if PPC_BOOK3S_64
+ default y if PPC_8xx && HUGETLB_PAGE
default n
config PPC_HAVE_PMU_SUPPORT
--
2.13.3
^ permalink raw reply related
* [PATCH v4 1/5] powerpc/mm/slice: Remove intermediate bitmap copy
From: Christophe Leroy @ 2018-02-10 12:54 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Scott Wood, aneesh.kumar
Cc: linux-kernel, linuxppc-dev, Nicholas Piggin
bitmap_or() and bitmap_andnot() can work properly with dst identical
to src1 or src2. There is no need of an intermediate result bitmap
that is copied back to dst in a second step.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
v2: New in v2
v3: patch moved up front of the serie to avoid ephemeral slice_bitmap_copy() function in following patch
v4: No change
arch/powerpc/mm/slice.c | 12 ++++--------
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 23ec2c5e3b78..98b53d48968f 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -388,21 +388,17 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
static inline void slice_or_mask(struct slice_mask *dst, struct slice_mask *src)
{
- DECLARE_BITMAP(result, SLICE_NUM_HIGH);
-
dst->low_slices |= src->low_slices;
- bitmap_or(result, dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
- bitmap_copy(dst->high_slices, result, SLICE_NUM_HIGH);
+ bitmap_or(dst->high_slices, dst->high_slices, src->high_slices,
+ SLICE_NUM_HIGH);
}
static inline void slice_andnot_mask(struct slice_mask *dst, struct slice_mask *src)
{
- DECLARE_BITMAP(result, SLICE_NUM_HIGH);
-
dst->low_slices &= ~src->low_slices;
- bitmap_andnot(result, dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
- bitmap_copy(dst->high_slices, result, SLICE_NUM_HIGH);
+ bitmap_andnot(dst->high_slices, dst->high_slices, src->high_slices,
+ SLICE_NUM_HIGH);
}
#ifdef CONFIG_PPC_64K_PAGES
--
2.13.3
^ permalink raw reply related
* [PATCH v4 5/5] powerpc/8xx: Increase number of slices to 64
From: Christophe Leroy @ 2018-02-10 12:54 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Scott Wood, aneesh.kumar
Cc: linux-kernel, linuxppc-dev, Nicholas Piggin
In-Reply-To: <01e8f783db8f4d4d41df91e0400a8634272b326f.1518226173.git.christophe.leroy@c-s.fr>
On the 8xx, the minimum slice size is the size of the area
covered by a single PMD entry, ie 4M in 4K pages mode and 64M in
16K pages mode.
This patch increases the number of slices from 16 to 64 on the 8xx.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v4: New
arch/powerpc/include/asm/nohash/32/slice.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/nohash/32/slice.h b/arch/powerpc/include/asm/nohash/32/slice.h
index bcb4924f7d22..3deaaea89473 100644
--- a/arch/powerpc/include/asm/nohash/32/slice.h
+++ b/arch/powerpc/include/asm/nohash/32/slice.h
@@ -4,7 +4,7 @@
#ifdef CONFIG_PPC_MM_SLICES
-#define SLICE_LOW_SHIFT 28
+#define SLICE_LOW_SHIFT 26 /* 64 slices */
#define SLICE_LOW_TOP (0x100000000ull)
#define SLICE_NUM_LOW (SLICE_LOW_TOP >> SLICE_LOW_SHIFT)
#define GET_LOW_SLICE_INDEX(addr) ((addr) >> SLICE_LOW_SHIFT)
--
2.13.3
^ permalink raw reply related
* [PATCH v4 2/5] powerpc/mm/slice: Enhance for supporting PPC32
From: Christophe Leroy @ 2018-02-10 12:54 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Scott Wood, aneesh.kumar
Cc: linux-kernel, linuxppc-dev, Nicholas Piggin
In-Reply-To: <01e8f783db8f4d4d41df91e0400a8634272b326f.1518226173.git.christophe.leroy@c-s.fr>
In preparation for the following patch which will fix an issue on
the 8xx by re-using the 'slices', this patch enhances the
'slices' implementation to support 32 bits CPUs.
On PPC32, the address space is limited to 4Gbytes, hence only the low
slices will be used.
This patch moves "slices" functions prototypes from page64.h to slice.h
The high slices use bitmaps. As bitmap functions are not prepared to
handling bitmaps of size 0, the bitmap_xxx() calls are wrapped into
slice_bitmap_xxx() functions which will void on PPC32
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v2: First patch of v1 serie split in two parts ; added slice_bitmap_xxx() macros.
v3: Moving slice related stuff in slice.h and slice_32/64.h
slice_bitmap_xxx() are now static inline functions and platform dependent
SLICE_LOW_TOP declared ull on PPC32 with correct casts allows to keep it 0x100000000
v4: Moved slice_32.h and slice_64.h to respective subarch dirs
Moved somes #ifdefs from asm/slice.h to respective subarch slice.h
SLICE_LOW_ details distributed in repective subarch slices allthough they are identical for the moment
arch/powerpc/include/asm/book3s/64/slice.h | 79 ++++++++++++++++++++++++++++++
arch/powerpc/include/asm/nohash/32/slice.h | 65 ++++++++++++++++++++++++
arch/powerpc/include/asm/nohash/64/slice.h | 12 +++++
arch/powerpc/include/asm/page.h | 1 +
arch/powerpc/include/asm/page_64.h | 59 ----------------------
arch/powerpc/include/asm/slice.h | 42 ++++++++++++++++
arch/powerpc/mm/slice.c | 38 ++++++++------
7 files changed, 221 insertions(+), 75 deletions(-)
create mode 100644 arch/powerpc/include/asm/book3s/64/slice.h
create mode 100644 arch/powerpc/include/asm/nohash/32/slice.h
create mode 100644 arch/powerpc/include/asm/nohash/64/slice.h
create mode 100644 arch/powerpc/include/asm/slice.h
diff --git a/arch/powerpc/include/asm/book3s/64/slice.h b/arch/powerpc/include/asm/book3s/64/slice.h
new file mode 100644
index 000000000000..f9a2c8bd7a77
--- /dev/null
+++ b/arch/powerpc/include/asm/book3s/64/slice.h
@@ -0,0 +1,79 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_POWERPC_BOOK3S_64_SLICE_H
+#define _ASM_POWERPC_BOOK3S_64_SLICE_H
+
+#ifdef CONFIG_PPC_MM_SLICES
+
+#define SLICE_LOW_SHIFT 28
+#define SLICE_LOW_TOP (0x100000000ul)
+#define SLICE_NUM_LOW (SLICE_LOW_TOP >> SLICE_LOW_SHIFT)
+#define GET_LOW_SLICE_INDEX(addr) ((addr) >> SLICE_LOW_SHIFT)
+
+#define SLICE_HIGH_SHIFT 40
+#define SLICE_NUM_HIGH (H_PGTABLE_RANGE >> SLICE_HIGH_SHIFT)
+#define GET_HIGH_SLICE_INDEX(addr) ((addr) >> SLICE_HIGH_SHIFT)
+
+#ifndef __ASSEMBLY__
+
+#include <linux/bitmap.h>
+
+static inline void slice_bitmap_zero(unsigned long *dst, unsigned int nbits)
+{
+ bitmap_zero(dst, nbits);
+}
+
+static inline int slice_bitmap_and(unsigned long *dst,
+ const unsigned long *src1,
+ const unsigned long *src2,
+ unsigned int nbits)
+{
+ return bitmap_and(dst, src1, src2, nbits);
+}
+
+static inline void slice_bitmap_or(unsigned long *dst,
+ const unsigned long *src1,
+ const unsigned long *src2,
+ unsigned int nbits)
+{
+ bitmap_or(dst, src1, src2, nbits);
+}
+
+static inline int slice_bitmap_andnot(unsigned long *dst,
+ const unsigned long *src1,
+ const unsigned long *src2,
+ unsigned int nbits)
+{
+ return bitmap_andnot(dst, src1, src2, nbits);
+}
+
+static inline int slice_bitmap_equal(const unsigned long *src1,
+ const unsigned long *src2,
+ unsigned int nbits)
+{
+ return bitmap_equal(src1, src2, nbits);
+}
+
+static inline int slice_bitmap_empty(const unsigned long *src, unsigned nbits)
+{
+ return bitmap_empty(src, nbits);
+}
+
+static inline void slice_bitmap_set(unsigned long *map, unsigned int start,
+ unsigned int nbits)
+{
+ bitmap_set(map, start, nbits);
+}
+#endif /* __ASSEMBLY__ */
+
+#else /* CONFIG_PPC_MM_SLICES */
+
+#define get_slice_psize(mm, addr) ((mm)->context.user_psize)
+#define slice_set_user_psize(mm, psize) \
+do { \
+ (mm)->context.user_psize = (psize); \
+ (mm)->context.sllp = SLB_VSID_USER | mmu_psize_defs[(psize)].sllp; \
+} while (0)
+
+#endif /* CONFIG_PPC_MM_SLICES */
+
+#endif /* _ASM_POWERPC_BOOK3S_64_SLICE_H */
diff --git a/arch/powerpc/include/asm/nohash/32/slice.h b/arch/powerpc/include/asm/nohash/32/slice.h
new file mode 100644
index 000000000000..bcb4924f7d22
--- /dev/null
+++ b/arch/powerpc/include/asm/nohash/32/slice.h
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_POWERPC_NOHASH_32_SLICE_H
+#define _ASM_POWERPC_NOHASH_32_SLICE_H
+
+#ifdef CONFIG_PPC_MM_SLICES
+
+#define SLICE_LOW_SHIFT 28
+#define SLICE_LOW_TOP (0x100000000ull)
+#define SLICE_NUM_LOW (SLICE_LOW_TOP >> SLICE_LOW_SHIFT)
+#define GET_LOW_SLICE_INDEX(addr) ((addr) >> SLICE_LOW_SHIFT)
+
+#define SLICE_HIGH_SHIFT 0
+#define SLICE_NUM_HIGH 0ul
+#define GET_HIGH_SLICE_INDEX(addr) (addr & 0)
+
+#ifndef __ASSEMBLY__
+
+static inline void slice_bitmap_zero(unsigned long *dst, unsigned int nbits)
+{
+}
+
+static inline int slice_bitmap_and(unsigned long *dst,
+ const unsigned long *src1,
+ const unsigned long *src2,
+ unsigned int nbits)
+{
+ return 0;
+}
+
+static inline void slice_bitmap_or(unsigned long *dst,
+ const unsigned long *src1,
+ const unsigned long *src2,
+ unsigned int nbits)
+{
+}
+
+static inline int slice_bitmap_andnot(unsigned long *dst,
+ const unsigned long *src1,
+ const unsigned long *src2,
+ unsigned int nbits)
+{
+ return 0;
+}
+
+static inline int slice_bitmap_equal(const unsigned long *src1,
+ const unsigned long *src2,
+ unsigned int nbits)
+{
+ return 1;
+}
+
+static inline int slice_bitmap_empty(const unsigned long *src, unsigned nbits)
+{
+ return 1;
+}
+
+static inline void slice_bitmap_set(unsigned long *map, unsigned int start,
+ unsigned int nbits)
+{
+}
+#endif /* __ASSEMBLY__ */
+
+#endif /* CONFIG_PPC_MM_SLICES */
+
+#endif /* _ASM_POWERPC_NOHASH_32_SLICE_H */
diff --git a/arch/powerpc/include/asm/nohash/64/slice.h b/arch/powerpc/include/asm/nohash/64/slice.h
new file mode 100644
index 000000000000..ad0d6e3cc1c5
--- /dev/null
+++ b/arch/powerpc/include/asm/nohash/64/slice.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_POWERPC_NOHASH_64_SLICE_H
+#define _ASM_POWERPC_NOHASH_64_SLICE_H
+
+#ifdef CONFIG_PPC_64K_PAGES
+#define get_slice_psize(mm, addr) MMU_PAGE_64K
+#else /* CONFIG_PPC_64K_PAGES */
+#define get_slice_psize(mm, addr) MMU_PAGE_4K
+#endif /* !CONFIG_PPC_64K_PAGES */
+#define slice_set_user_psize(mm, psize) do { BUG(); } while (0)
+
+#endif /* _ASM_POWERPC_NOHASH_64_SLICE_H */
diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h
index 8da5d4c1cab2..d5f1c41b7dba 100644
--- a/arch/powerpc/include/asm/page.h
+++ b/arch/powerpc/include/asm/page.h
@@ -344,5 +344,6 @@ typedef struct page *pgtable_t;
#include <asm-generic/memory_model.h>
#endif /* __ASSEMBLY__ */
+#include <asm/slice.h>
#endif /* _ASM_POWERPC_PAGE_H */
diff --git a/arch/powerpc/include/asm/page_64.h b/arch/powerpc/include/asm/page_64.h
index 56234c6fcd61..af04acdb873f 100644
--- a/arch/powerpc/include/asm/page_64.h
+++ b/arch/powerpc/include/asm/page_64.h
@@ -86,65 +86,6 @@ extern u64 ppc64_pft_size;
#endif /* __ASSEMBLY__ */
-#ifdef CONFIG_PPC_MM_SLICES
-
-#define SLICE_LOW_SHIFT 28
-#define SLICE_HIGH_SHIFT 40
-
-#define SLICE_LOW_TOP (0x100000000ul)
-#define SLICE_NUM_LOW (SLICE_LOW_TOP >> SLICE_LOW_SHIFT)
-#define SLICE_NUM_HIGH (H_PGTABLE_RANGE >> SLICE_HIGH_SHIFT)
-
-#define GET_LOW_SLICE_INDEX(addr) ((addr) >> SLICE_LOW_SHIFT)
-#define GET_HIGH_SLICE_INDEX(addr) ((addr) >> SLICE_HIGH_SHIFT)
-
-#ifndef __ASSEMBLY__
-struct mm_struct;
-
-extern unsigned long slice_get_unmapped_area(unsigned long addr,
- unsigned long len,
- unsigned long flags,
- unsigned int psize,
- int topdown);
-
-extern unsigned int get_slice_psize(struct mm_struct *mm,
- unsigned long addr);
-
-extern void slice_set_user_psize(struct mm_struct *mm, unsigned int psize);
-extern void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
- unsigned long len, unsigned int psize);
-
-#endif /* __ASSEMBLY__ */
-#else
-#define slice_init()
-#ifdef CONFIG_PPC_BOOK3S_64
-#define get_slice_psize(mm, addr) ((mm)->context.user_psize)
-#define slice_set_user_psize(mm, psize) \
-do { \
- (mm)->context.user_psize = (psize); \
- (mm)->context.sllp = SLB_VSID_USER | mmu_psize_defs[(psize)].sllp; \
-} while (0)
-#else /* !CONFIG_PPC_BOOK3S_64 */
-#ifdef CONFIG_PPC_64K_PAGES
-#define get_slice_psize(mm, addr) MMU_PAGE_64K
-#else /* CONFIG_PPC_64K_PAGES */
-#define get_slice_psize(mm, addr) MMU_PAGE_4K
-#endif /* !CONFIG_PPC_64K_PAGES */
-#define slice_set_user_psize(mm, psize) do { BUG(); } while(0)
-#endif /* CONFIG_PPC_BOOK3S_64 */
-
-#define slice_set_range_psize(mm, start, len, psize) \
- slice_set_user_psize((mm), (psize))
-#endif /* CONFIG_PPC_MM_SLICES */
-
-#ifdef CONFIG_HUGETLB_PAGE
-
-#ifdef CONFIG_PPC_MM_SLICES
-#define HAVE_ARCH_HUGETLB_UNMAPPED_AREA
-#endif
-
-#endif /* !CONFIG_HUGETLB_PAGE */
-
#define VM_DATA_DEFAULT_FLAGS \
(is_32bit_task() ? \
VM_DATA_DEFAULT_FLAGS32 : VM_DATA_DEFAULT_FLAGS64)
diff --git a/arch/powerpc/include/asm/slice.h b/arch/powerpc/include/asm/slice.h
new file mode 100644
index 000000000000..172711fadb1c
--- /dev/null
+++ b/arch/powerpc/include/asm/slice.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_POWERPC_SLICE_H
+#define _ASM_POWERPC_SLICE_H
+
+#ifdef CONFIG_PPC_BOOK3S_64
+#include <asm/book3s/64/slice.h>
+#elif defined(CONFIG_PPC64)
+#include <asm/nohash/64/slice.h>
+#elif defined(CONFIG_PPC_MMU_NOHASH)
+#include <asm/nohash/32/slice.h>
+#endif
+
+#ifdef CONFIG_PPC_MM_SLICES
+
+#ifdef CONFIG_HUGETLB_PAGE
+#define HAVE_ARCH_HUGETLB_UNMAPPED_AREA
+#endif
+#define HAVE_ARCH_UNMAPPED_AREA
+#define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN
+
+#ifndef __ASSEMBLY__
+
+struct mm_struct;
+
+unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
+ unsigned long flags, unsigned int psize,
+ int topdown);
+
+unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr);
+
+void slice_set_user_psize(struct mm_struct *mm, unsigned int psize);
+void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
+ unsigned long len, unsigned int psize);
+#endif /* __ASSEMBLY__ */
+
+#else /* CONFIG_PPC_MM_SLICES */
+
+#define slice_set_range_psize(mm, start, len, psize) \
+ slice_set_user_psize((mm), (psize))
+#endif /* CONFIG_PPC_MM_SLICES */
+
+#endif /* _ASM_POWERPC_SLICE_H */
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 98b53d48968f..549704dfa777 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -73,10 +73,11 @@ static void slice_range_to_mask(unsigned long start, unsigned long len,
unsigned long end = start + len - 1;
ret->low_slices = 0;
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
+ slice_bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
if (start < SLICE_LOW_TOP) {
- unsigned long mend = min(end, (SLICE_LOW_TOP - 1));
+ unsigned long mend = min(end,
+ (unsigned long)(SLICE_LOW_TOP - 1));
ret->low_slices = (1u << (GET_LOW_SLICE_INDEX(mend) + 1))
- (1u << GET_LOW_SLICE_INDEX(start));
@@ -87,7 +88,7 @@ static void slice_range_to_mask(unsigned long start, unsigned long len,
unsigned long align_end = ALIGN(end, (1UL << SLICE_HIGH_SHIFT));
unsigned long count = GET_HIGH_SLICE_INDEX(align_end) - start_index;
- bitmap_set(ret->high_slices, start_index, count);
+ slice_bitmap_set(ret->high_slices, start_index, count);
}
}
@@ -113,11 +114,13 @@ static int slice_high_has_vma(struct mm_struct *mm, unsigned long slice)
unsigned long start = slice << SLICE_HIGH_SHIFT;
unsigned long end = start + (1ul << SLICE_HIGH_SHIFT);
+#ifdef CONFIG_PPC64
/* Hack, so that each addresses is controlled by exactly one
* of the high or low area bitmaps, the first high area starts
* at 4GB, not 0 */
if (start == 0)
start = SLICE_LOW_TOP;
+#endif
return !slice_area_is_free(mm, start, end - start);
}
@@ -128,7 +131,7 @@ static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
unsigned long i;
ret->low_slices = 0;
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
+ slice_bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
for (i = 0; i < SLICE_NUM_LOW; i++)
if (!slice_low_has_vma(mm, i))
@@ -151,7 +154,7 @@ static void slice_mask_for_size(struct mm_struct *mm, int psize, struct slice_ma
u64 lpsizes;
ret->low_slices = 0;
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
+ slice_bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
lpsizes = mm->context.low_slices_psize;
for (i = 0; i < SLICE_NUM_LOW; i++)
@@ -180,15 +183,16 @@ static int slice_check_fit(struct mm_struct *mm,
*/
unsigned long slice_count = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
- bitmap_and(result, mask.high_slices,
- available.high_slices, slice_count);
+ slice_bitmap_and(result, mask.high_slices, available.high_slices,
+ slice_count);
return (mask.low_slices & available.low_slices) == mask.low_slices &&
- bitmap_equal(result, mask.high_slices, slice_count);
+ slice_bitmap_equal(result, mask.high_slices, slice_count);
}
static void slice_flush_segments(void *parm)
{
+#ifdef CONFIG_PPC64
struct mm_struct *mm = parm;
unsigned long flags;
@@ -200,6 +204,7 @@ static void slice_flush_segments(void *parm)
local_irq_save(flags);
slb_flush_and_rebolt();
local_irq_restore(flags);
+#endif
}
static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psize)
@@ -389,16 +394,16 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
static inline void slice_or_mask(struct slice_mask *dst, struct slice_mask *src)
{
dst->low_slices |= src->low_slices;
- bitmap_or(dst->high_slices, dst->high_slices, src->high_slices,
- SLICE_NUM_HIGH);
+ slice_bitmap_or(dst->high_slices, dst->high_slices, src->high_slices,
+ SLICE_NUM_HIGH);
}
static inline void slice_andnot_mask(struct slice_mask *dst, struct slice_mask *src)
{
dst->low_slices &= ~src->low_slices;
- bitmap_andnot(dst->high_slices, dst->high_slices, src->high_slices,
- SLICE_NUM_HIGH);
+ slice_bitmap_andnot(dst->high_slices, dst->high_slices,
+ src->high_slices, SLICE_NUM_HIGH);
}
#ifdef CONFIG_PPC_64K_PAGES
@@ -446,14 +451,14 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* init different masks
*/
mask.low_slices = 0;
- bitmap_zero(mask.high_slices, SLICE_NUM_HIGH);
+ slice_bitmap_zero(mask.high_slices, SLICE_NUM_HIGH);
/* silence stupid warning */;
potential_mask.low_slices = 0;
- bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
+ slice_bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
compat_mask.low_slices = 0;
- bitmap_zero(compat_mask.high_slices, SLICE_NUM_HIGH);
+ slice_bitmap_zero(compat_mask.high_slices, SLICE_NUM_HIGH);
/* Sanity checks */
BUG_ON(mm->task_size == 0);
@@ -591,7 +596,8 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
convert:
slice_andnot_mask(&mask, &good_mask);
slice_andnot_mask(&mask, &compat_mask);
- if (mask.low_slices || !bitmap_empty(mask.high_slices, SLICE_NUM_HIGH)) {
+ if (mask.low_slices ||
+ !slice_bitmap_empty(mask.high_slices, SLICE_NUM_HIGH)) {
slice_convert(mm, mask, psize);
if (psize > MMU_PAGE_BASE)
on_each_cpu(slice_flush_segments, mm, 1);
--
2.13.3
^ permalink raw reply related
* [RFC PATCH 5/5] powerpc/mm/hash64: Increase the VA range
From: Aneesh Kumar K.V @ 2018-02-10 10:56 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <20180210105643.11857-1-aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/64/hash-64k.h | 2 +-
arch/powerpc/include/asm/processor.h | 9 ++++++++-
arch/powerpc/mm/init_64.c | 6 ------
arch/powerpc/mm/pgtable_64.c | 5 -----
4 files changed, 9 insertions(+), 13 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/hash-64k.h b/arch/powerpc/include/asm/book3s/64/hash-64k.h
index d21ff0372f19..3377de7d6207 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-64k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-64k.h
@@ -4,7 +4,7 @@
#define H_PTE_INDEX_SIZE 8
#define H_PMD_INDEX_SIZE 10
-#define H_PUD_INDEX_SIZE 7
+#define H_PUD_INDEX_SIZE 10
#define H_PGD_INDEX_SIZE 8
/*
* No of address bits below which we use the default context
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 01299cdc9806..73050cb412b4 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -109,6 +109,13 @@ void release_thread(struct task_struct *);
#define TASK_SIZE_64TB (0x0000400000000000UL)
#define TASK_SIZE_128TB (0x0000800000000000UL)
#define TASK_SIZE_512TB (0x0002000000000000UL)
+#define TASK_SIZE_1PB (0x0004000000000000UL)
+#define TASK_SIZE_2PB (0x0008000000000000UL)
+/*
+ * With 52 bits in the address we can support
+ * upto 4PB of range.
+ */
+#define TASK_SIZE_4PB (0x0010000000000000UL)
/*
* For now 512TB is only supported with book3s and 64K linux page size.
@@ -117,7 +124,7 @@ void release_thread(struct task_struct *);
/*
* Max value currently used:
*/
-#define TASK_SIZE_USER64 TASK_SIZE_512TB
+#define TASK_SIZE_USER64 TASK_SIZE_4PB
#define DEFAULT_MAP_WINDOW_USER64 TASK_SIZE_128TB
#else
#define TASK_SIZE_USER64 TASK_SIZE_64TB
diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index f6eb7e8f4c93..624ef8e3e064 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -68,12 +68,6 @@
#include "mmu_decl.h"
-#ifdef CONFIG_PPC_BOOK3S_64
-#if H_PGTABLE_RANGE > USER_VSID_RANGE
-#warning Limited user VSID range means pagetable space is wasted
-#endif
-#endif /* CONFIG_PPC_BOOK3S_64 */
-
phys_addr_t memstart_addr = ~0;
EXPORT_SYMBOL_GPL(memstart_addr);
phys_addr_t kernstart_addr;
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index a0f8928c0b86..8c9f98771589 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -57,11 +57,6 @@
#include "mmu_decl.h"
-#ifdef CONFIG_PPC_BOOK3S_64
-#if TASK_SIZE_USER64 > (1UL << (ESID_BITS + SID_SHIFT))
-#error TASK_SIZE_USER64 exceeds user VSID range
-#endif
-#endif
#ifdef CONFIG_PPC_BOOK3S_64
/*
--
2.14.3
^ permalink raw reply related
* [RFC PATCH 4/5] powerpc/mm: Add support for handling > 512TB address in SLB miss
From: Aneesh Kumar K.V @ 2018-02-10 10:56 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <20180210105643.11857-1-aneesh.kumar@linux.vnet.ibm.com>
For address above 512TB we allocate additonal mmu context. To make it all
easy address above 512TB is handled with IR/DR=1 and with stack frame setup.
We do the additonal context allocation in SLB miss handler. If the context is
not allocated, we enable interrupts and allocate the context and retry the
access which will again result in a SLB miss.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/64/hash-4k.h | 6 ++
arch/powerpc/include/asm/book3s/64/hash-64k.h | 5 +
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 6 +-
arch/powerpc/include/asm/book3s/64/mmu.h | 24 +++++
arch/powerpc/kernel/exceptions-64s.S | 12 ++-
arch/powerpc/mm/copro_fault.c | 2 +-
arch/powerpc/mm/hash_utils_64.c | 4 +-
arch/powerpc/mm/mmu_context_book3s64.c | 17 ++-
arch/powerpc/mm/pgtable-hash64.c | 2 +-
arch/powerpc/mm/slb.c | 144 ++++++++++++++++++++++++++
arch/powerpc/mm/slb_low.S | 6 +-
arch/powerpc/mm/tlb_hash64.c | 2 +-
12 files changed, 215 insertions(+), 15 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/hash-4k.h b/arch/powerpc/include/asm/book3s/64/hash-4k.h
index 8a91f2475ea3..fb1a75df23e3 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-4k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-4k.h
@@ -11,6 +11,12 @@
#define H_PUD_INDEX_SIZE 9
#define H_PGD_INDEX_SIZE 9
+/*
+ * No of address bits below which we use the default context
+ * for slb allocation. For 4k this is 64TB.
+ */
+#define H_BITS_FIRST_CONTEXT 46
+
#ifndef __ASSEMBLY__
#define H_PTE_TABLE_SIZE (sizeof(pte_t) << H_PTE_INDEX_SIZE)
#define H_PMD_TABLE_SIZE (sizeof(pmd_t) << H_PMD_INDEX_SIZE)
diff --git a/arch/powerpc/include/asm/book3s/64/hash-64k.h b/arch/powerpc/include/asm/book3s/64/hash-64k.h
index 1dace73e1d12..d21ff0372f19 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-64k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-64k.h
@@ -6,6 +6,11 @@
#define H_PMD_INDEX_SIZE 10
#define H_PUD_INDEX_SIZE 7
#define H_PGD_INDEX_SIZE 8
+/*
+ * No of address bits below which we use the default context
+ * for slb allocation. For 64k this is 512TB.
+ */
+#define H_BITS_FIRST_CONTEXT 49
/*
* 64k aligned address free up few of the lower bits of RPN for us
diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
index 50ed64fba4ae..8ee83f6e9c84 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
@@ -691,8 +691,8 @@ static inline int user_segment_size(unsigned long addr)
return MMU_SEGSIZE_256M;
}
-static inline unsigned long get_vsid(unsigned long context, unsigned long ea,
- int ssize)
+static inline unsigned long __get_vsid(unsigned long context, unsigned long ea,
+ int ssize)
{
unsigned long va_bits = VA_BITS;
unsigned long vsid_bits;
@@ -744,7 +744,7 @@ static inline unsigned long get_kernel_vsid(unsigned long ea, int ssize)
*/
context = (ea >> 60) - KERNEL_REGION_CONTEXT_OFFSET;
- return get_vsid(context, ea, ssize);
+ return __get_vsid(context, ea, ssize);
}
unsigned htab_shift_for_mem_size(unsigned long mem_size);
diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
index 0abeb0e2d616..ebd2f62050ee 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu.h
@@ -82,6 +82,12 @@ struct spinlock;
typedef struct {
mm_context_id_t id;
+ /*
+ * One context for each 512TB above the first 512TB.
+ * First 512TB context is saved in id and is also used
+ * as PIDR.
+ */
+ mm_context_id_t extended_id[(TASK_SIZE_USER64/TASK_SIZE_512TB) - 1];
u16 user_psize; /* page size index */
/* Number of bits in the mm_cpumask */
@@ -174,5 +180,23 @@ extern void radix_init_pseries(void);
static inline void radix_init_pseries(void) { };
#endif
+static inline int get_esid_context(mm_context_t *ctx, unsigned long ea)
+{
+ int index = ea >> H_BITS_FIRST_CONTEXT;
+
+ if (index == 0)
+ return ctx->id;
+ return ctx->extended_id[--index];
+}
+
+static inline unsigned long get_user_vsid(mm_context_t *ctx,
+ unsigned long ea, int ssize)
+{
+ unsigned long context = get_esid_context(ctx, ea);
+
+ return __get_vsid(context, ea, ssize);
+}
+
+
#endif /* __ASSEMBLY__ */
#endif /* _ASM_POWERPC_BOOK3S_64_MMU_H_ */
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 9e6882bdc526..d8188d52327c 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -620,8 +620,12 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX)
ld r10,PACA_EXSLB+EX_LR(r13)
lwz r9,PACA_EXSLB+EX_CCR(r13) /* get saved CR */
mtlr r10
+ /*
+ * Large address, check whether we have to allocate new
+ * contexts.
+ */
+ beq- 8f
- beq- 8f /* if bad address, make full stack frame */
bne- cr5,2f /* if unrecoverable exception, oops */
@@ -685,7 +689,7 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX)
mr r3,r12
mfspr r11,SPRN_SRR0
mfspr r12,SPRN_SRR1
- LOAD_HANDLER(r10,bad_addr_slb)
+ LOAD_HANDLER(r10, multi_context_slb)
mtspr SPRN_SRR0,r10
ld r10,PACAKMSR(r13)
mtspr SPRN_SRR1,r10
@@ -700,7 +704,7 @@ EXC_COMMON_BEGIN(unrecov_slb)
bl unrecoverable_exception
b 1b
-EXC_COMMON_BEGIN(bad_addr_slb)
+EXC_COMMON_BEGIN(multi_context_slb)
EXCEPTION_PROLOG_COMMON(0x380, PACA_EXSLB)
RECONCILE_IRQ_STATE(r10, r11)
ld r3, PACA_EXSLB+EX_DAR(r13)
@@ -710,7 +714,7 @@ EXC_COMMON_BEGIN(bad_addr_slb)
std r10, _TRAP(r1)
2: bl save_nvgprs
addi r3, r1, STACK_FRAME_OVERHEAD
- bl slb_miss_bad_addr
+ bl handle_multi_context_slb_miss
b ret_from_except
EXC_REAL_BEGIN(hardware_interrupt, 0x500, 0x100)
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index 697b70ad1195..7d0945bd3a61 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -112,7 +112,7 @@ int copro_calculate_slb(struct mm_struct *mm, u64 ea, struct copro_slb *slb)
return 1;
psize = get_slice_psize(mm, ea);
ssize = user_segment_size(ea);
- vsid = get_vsid(mm->context.id, ea, ssize);
+ vsid = get_user_vsid(&mm->context, ea, ssize);
vsidkey = SLB_VSID_USER;
break;
case VMALLOC_REGION_ID:
diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
index cf290d415dcd..d6fdf20cdd7b 100644
--- a/arch/powerpc/mm/hash_utils_64.c
+++ b/arch/powerpc/mm/hash_utils_64.c
@@ -1262,7 +1262,7 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea,
}
psize = get_slice_psize(mm, ea);
ssize = user_segment_size(ea);
- vsid = get_vsid(mm->context.id, ea, ssize);
+ vsid = get_user_vsid(&mm->context, ea, ssize);
break;
case VMALLOC_REGION_ID:
vsid = get_kernel_vsid(ea, mmu_kernel_ssize);
@@ -1527,7 +1527,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea,
/* Get VSID */
ssize = user_segment_size(ea);
- vsid = get_vsid(mm->context.id, ea, ssize);
+ vsid = get_user_vsid(&mm->context, ea, ssize);
if (!vsid)
return;
/*
diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c
index 929d9ef7083f..fbcba7337354 100644
--- a/arch/powerpc/mm/mmu_context_book3s64.c
+++ b/arch/powerpc/mm/mmu_context_book3s64.c
@@ -185,6 +185,19 @@ void __destroy_context(int context_id)
}
EXPORT_SYMBOL_GPL(__destroy_context);
+void destroy_extended_context(mm_context_t *ctx)
+{
+ int index, context_id;
+
+ spin_lock(&mmu_context_lock);
+ for (index = 1; index < (TASK_SIZE_USER64/TASK_SIZE_512TB); index++) {
+ context_id = ctx->extended_id[index - 1];
+ if (context_id != 0)
+ ida_remove(&mmu_context_ida, context_id);
+ }
+ spin_unlock(&mmu_context_lock);
+}
+
#ifdef CONFIG_PPC_64K_PAGES
static void destroy_pagetable_page(struct mm_struct *mm)
{
@@ -220,8 +233,10 @@ void destroy_context(struct mm_struct *mm)
#endif
if (radix_enabled())
WARN_ON(process_tb[mm->context.id].prtb0 != 0);
- else
+ else {
subpage_prot_free(mm);
+ destroy_extended_context(&mm->context);
+ }
destroy_pagetable_page(mm);
__destroy_context(mm->context.id);
mm->context.id = MMU_NO_CONTEXT;
diff --git a/arch/powerpc/mm/pgtable-hash64.c b/arch/powerpc/mm/pgtable-hash64.c
index ec277913e01b..2e01794709ad 100644
--- a/arch/powerpc/mm/pgtable-hash64.c
+++ b/arch/powerpc/mm/pgtable-hash64.c
@@ -342,7 +342,7 @@ void hpte_do_hugepage_flush(struct mm_struct *mm, unsigned long addr,
if (!is_kernel_addr(addr)) {
ssize = user_segment_size(addr);
- vsid = get_vsid(mm->context.id, addr, ssize);
+ vsid = get_user_vsid(&mm->context, addr, ssize);
WARN_ON(vsid == 0);
} else {
vsid = get_kernel_vsid(addr, mmu_kernel_ssize);
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 13cfe413b40d..31e7ad0608f0 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -23,6 +23,7 @@
#include <asm/smp.h>
#include <linux/compiler.h>
#include <linux/mm_types.h>
+#include <linux/context_tracking.h>
#include <asm/udbg.h>
#include <asm/code-patching.h>
@@ -340,3 +341,146 @@ void slb_initialize(void)
asm volatile("isync":::"memory");
}
+
+/*
+ * Only handle insert of 1TB slb entries.
+ */
+static void insert_slb_entry(unsigned long vsid, unsigned long ea,
+ int bpsize, int ssize)
+{
+ int slb_cache_index;
+ unsigned long flags;
+ enum slb_index index;
+ unsigned long vsid_data, esid_data;
+
+ /*
+ * We are irq disabled, hence should be safe
+ * to access PACA.
+ */
+ index = get_paca()->stab_rr;
+ /*
+ * simple round roubin replacement of slb.
+ */
+ if (index < mmu_slb_size)
+ index++;
+ else
+ index = SLB_NUM_BOLTED;
+ get_paca()->stab_rr = index;
+
+ flags = SLB_VSID_USER | mmu_psize_defs[bpsize].sllp;
+ vsid_data = (vsid << SLB_VSID_SHIFT_1T) | flags |
+ ((unsigned long) ssize << SLB_VSID_SSIZE_SHIFT);
+ esid_data = mk_esid_data(ea, mmu_highuser_ssize, index);
+
+ asm volatile("slbmte %0, %1" : : "r" (vsid_data), "r" (esid_data)
+ : "memory");
+ /*
+ * Now update slb cache entries
+ */
+ slb_cache_index = get_paca()->slb_cache_ptr;
+ if (slb_cache_index < SLB_CACHE_ENTRIES) {
+ /*
+ * We have space in slb cache for optimized switch_slb().
+ * Top 36 bits from esid_data as per ISA
+ */
+ get_paca()->slb_cache[slb_cache_index++] = esid_data >> 28;
+ }
+ /*
+ * if we are full, just increment and return.
+ */
+ get_paca()->slb_cache_ptr++;
+ return;
+}
+
+static void alloc_extended_context(struct mm_struct *mm, unsigned long ea)
+{
+ int context_id;
+
+ int index = (ea >> H_BITS_FIRST_CONTEXT) - 1;
+
+ /*
+ * FIXME!! Another lock than mmap_sem?
+ */
+ down_read(&mm->mmap_sem);
+ context_id = hash__alloc_context_id();
+ if (context_id < 0) {
+ up_read(&mm->mmap_sem);
+ pagefault_out_of_memory();
+ return;
+ }
+ /* Check for parallel allocation after holding lock */
+ if (mm->context.extended_id[index] < MIN_USER_CONTEXT)
+ mm->context.extended_id[index] = context_id;
+ else
+ __destroy_context(context_id);
+ up_read(&mm->mmap_sem);
+ return;
+}
+
+static void __handle_multi_context_slb_miss(struct pt_regs *regs,
+ unsigned long ea)
+{
+ int context, bpsize;
+ unsigned long vsid;
+ struct mm_struct *mm = current->mm;
+
+ context = get_esid_context(&mm->context, ea);
+ if (context < MIN_USER_CONTEXT) {
+ /*
+ * haven't allocated context yet for this range.
+ * Enable irq and allo context and return. We will
+ * take an slb miss on this again and come here with
+ * allocated context.
+ */
+ /* We restore the interrupt state now */
+ if (!arch_irq_disabled_regs(regs))
+ local_irq_enable();
+ return alloc_extended_context(mm, ea);
+ }
+ /*
+ * We are always above 1TB, hence use high user segment size.
+ */
+ vsid = __get_vsid(context, ea, mmu_highuser_ssize);
+ bpsize = get_slice_psize(mm, ea);
+
+ insert_slb_entry(vsid, ea, bpsize, mmu_highuser_ssize);
+ return;
+}
+
+/*
+ * exception_enter() handling? FIXME!!
+ */
+void handle_multi_context_slb_miss(struct pt_regs *regs)
+{
+ enum ctx_state prev_state = exception_enter();
+ unsigned long ea = regs->dar;
+
+ /*
+ * Kernel always runs with single context. Hence
+ * anything that request for multi context is
+ * considered bad slb request.
+ */
+ if (!user_mode(regs))
+ return bad_page_fault(regs, ea, SIGSEGV);
+
+ if (REGION_ID(ea) != USER_REGION_ID)
+ goto slb_bad_addr;
+ /*
+ * Are we beyound what the page table layout support ?
+ */
+ if ((ea & ~REGION_MASK) >= H_PGTABLE_RANGE)
+ goto slb_bad_addr;
+
+ /* Lower address should be handled by asm code */
+ if (ea <= (1UL << H_BITS_FIRST_CONTEXT))
+ goto slb_bad_addr;
+
+ __handle_multi_context_slb_miss(regs, ea);
+ exception_exit(prev_state);
+ return;
+
+slb_bad_addr:
+ _exception(SIGSEGV, regs, SEGV_BNDERR, ea);
+ exception_exit(prev_state);
+ return;
+}
diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S
index 2cf5ef3fc50d..425b4c5ec1e9 100644
--- a/arch/powerpc/mm/slb_low.S
+++ b/arch/powerpc/mm/slb_low.S
@@ -75,10 +75,12 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_68_BIT_VA)
*/
_GLOBAL(slb_allocate)
/*
- * check for bad kernel/user address
+ * Check for address range for which we need to handle multi context. For
+ * the default context we allocate the slb via the fast path. For large
+ * address we branch out to C-code and look at additional context allocated.
* (ea & ~REGION_MASK) >= PGTABLE_RANGE
*/
- rldicr. r9,r3,4,(63 - H_PGTABLE_EADDR_SIZE - 4)
+ rldicr. r9,r3,4,(63 - H_BITS_FIRST_CONTEXT - 4)
bne- 8f
srdi r9,r3,60 /* get region */
diff --git a/arch/powerpc/mm/tlb_hash64.c b/arch/powerpc/mm/tlb_hash64.c
index 881ebd53ffc2..08a69985cfca 100644
--- a/arch/powerpc/mm/tlb_hash64.c
+++ b/arch/powerpc/mm/tlb_hash64.c
@@ -84,7 +84,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr,
/* Build full vaddr */
if (!is_kernel_addr(addr)) {
ssize = user_segment_size(addr);
- vsid = get_vsid(mm->context.id, addr, ssize);
+ vsid = get_user_vsid(&mm->context, addr, ssize);
} else {
vsid = get_kernel_vsid(addr, mmu_kernel_ssize);
ssize = mmu_kernel_ssize;
--
2.14.3
^ permalink raw reply related
* [RFC PATCH 3/5] powerpc/mm/slice: Reduce the stack usage in slice_get_unmapped_area
From: Aneesh Kumar K.V @ 2018-02-10 10:56 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <20180210105643.11857-1-aneesh.kumar@linux.vnet.ibm.com>
This patch kill potential_mask and compat_mask variable and instead use tmp_mask
so that we can reduce the stack usage. This is required so that we can increase
the high_slices bitmap to a larger value.
The patch does result in extra computation in final stage, where it ends up
recomputing the compat mask again.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/mm/slice.c | 34 +++++++++++++++++-----------------
1 file changed, 17 insertions(+), 17 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 259bbda9a222..832c681c341a 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -413,8 +413,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
{
struct slice_mask mask;
struct slice_mask good_mask;
- struct slice_mask potential_mask;
- struct slice_mask compat_mask;
+ struct slice_mask tmp_mask;
int fixed = (flags & MAP_FIXED);
int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT);
unsigned long page_size = 1UL << pshift;
@@ -449,11 +448,8 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
bitmap_zero(mask.high_slices, SLICE_NUM_HIGH);
/* silence stupid warning */;
- potential_mask.low_slices = 0;
- bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
-
- compat_mask.low_slices = 0;
- bitmap_zero(compat_mask.high_slices, SLICE_NUM_HIGH);
+ tmp_mask.low_slices = 0;
+ bitmap_zero(tmp_mask.high_slices, SLICE_NUM_HIGH);
/* Sanity checks */
BUG_ON(mm->task_size == 0);
@@ -502,9 +498,11 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
/* If we support combo pages, we can allow 64k pages in 4k slices */
if (psize == MMU_PAGE_64K) {
- slice_mask_for_size(mm, MMU_PAGE_4K, &compat_mask, high_limit);
+ slice_mask_for_size(mm, MMU_PAGE_4K, &tmp_mask, high_limit);
if (fixed)
- slice_or_mask(&good_mask, &compat_mask);
+ slice_or_mask(&good_mask, &tmp_mask);
+
+ slice_print_mask("Mask for compat page size", tmp_mask);
}
#endif
/* First check hint if it's valid or if we have MAP_FIXED */
@@ -541,11 +539,11 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* We don't fit in the good mask, check what other slices are
* empty and thus can be converted
*/
- slice_mask_for_free(mm, &potential_mask, high_limit);
- slice_or_mask(&potential_mask, &good_mask);
- slice_print_mask(" potential", potential_mask);
+ slice_mask_for_free(mm, &tmp_mask, high_limit);
+ slice_or_mask(&tmp_mask, &good_mask);
+ slice_print_mask("Free area/potential ", tmp_mask);
- if ((addr != 0 || fixed) && slice_check_fit(mm, mask, potential_mask)) {
+ if ((addr != 0 || fixed) && slice_check_fit(mm, mask, tmp_mask)) {
slice_dbg(" fits potential !\n");
goto convert;
}
@@ -571,7 +569,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* Now let's see if we can find something in the existing slices
* for that size plus free slices
*/
- addr = slice_find_area(mm, len, potential_mask,
+ addr = slice_find_area(mm, len, tmp_mask,
psize, topdown, high_limit);
#ifdef CONFIG_PPC_64K_PAGES
@@ -585,9 +583,10 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* mask variable is free here. Use that for compat
* size mask.
*/
+ slice_mask_for_size(mm, MMU_PAGE_4K, &mask, high_limit);
/* retry the search with 4k-page slices included */
- slice_or_mask(&potential_mask, &compat_mask);
- addr = slice_find_area(mm, len, potential_mask,
+ slice_or_mask(&tmp_mask, &mask);
+ addr = slice_find_area(mm, len, tmp_mask,
psize, topdown, high_limit);
}
#endif
@@ -600,8 +599,9 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
slice_print_mask(" mask", mask);
convert:
+ slice_mask_for_size(mm, MMU_PAGE_4K, &tmp_mask, high_limit);
slice_andnot_mask(&mask, &good_mask);
- slice_andnot_mask(&mask, &compat_mask);
+ slice_andnot_mask(&mask, &tmp_mask);
if (mask.low_slices || !bitmap_empty(mask.high_slices, SLICE_NUM_HIGH)) {
slice_convert(mm, mask, psize);
if (psize > MMU_PAGE_BASE)
--
2.14.3
^ permalink raw reply related
* [RFC PATCH 2/5] powerpc/mm/slice: Update documentation in the file.
From: Aneesh Kumar K.V @ 2018-02-10 10:56 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <20180210105643.11857-1-aneesh.kumar@linux.vnet.ibm.com>
We will make code changes in the next patch. To make the review easier split
the documentation update in to a seperate patch.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/mm/slice.c | 27 +++++++++++++++++++--------
1 file changed, 19 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 98b53d48968f..259bbda9a222 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -478,7 +478,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* already
*/
slice_mask_for_size(mm, psize, &good_mask, high_limit);
- slice_print_mask(" good_mask", good_mask);
+ slice_print_mask("Mask for page size", good_mask);
/*
* Here "good" means slices that are already the right page size,
@@ -507,15 +507,17 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
slice_or_mask(&good_mask, &compat_mask);
}
#endif
-
/* First check hint if it's valid or if we have MAP_FIXED */
if (addr != 0 || fixed) {
- /* Build a mask for the requested range */
+ /*
+ * Build a mask for the requested range
+ */
slice_range_to_mask(addr, len, &mask);
- slice_print_mask(" mask", mask);
+ slice_print_mask("Request range mask", mask);
- /* Check if we fit in the good mask. If we do, we just return,
- * nothing else to do
+ /*
+ * Check if we fit in the good mask. If we do, we just
+ * return, nothing else to do
*/
if (slice_check_fit(mm, mask, good_mask)) {
slice_dbg(" fits good !\n");
@@ -553,8 +555,8 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
return -EBUSY;
slice_dbg(" search...\n");
-
- /* If we had a hint that didn't work out, see if we can fit
+ /*
+ * If we had a hint that didn't work out, see if we can fit
* anywhere in the good area.
*/
if (addr) {
@@ -573,7 +575,16 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
psize, topdown, high_limit);
#ifdef CONFIG_PPC_64K_PAGES
+ /*
+ * If we didn't request for fixed mapping, we never looked at
+ * compat area. Now that we are not finding space, let's look
+ * at the 4K slice also.
+ */
if (addr == -ENOMEM && psize == MMU_PAGE_64K) {
+ /*
+ * mask variable is free here. Use that for compat
+ * size mask.
+ */
/* retry the search with 4k-page slices included */
slice_or_mask(&potential_mask, &compat_mask);
addr = slice_find_area(mm, len, potential_mask,
--
2.14.3
^ permalink raw reply related
* [RFC PATCH 1/5] powerpc: Don't do runtime futex_cmpxchg test
From: Aneesh Kumar K.V @ 2018-02-10 10:56 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <20180210105643.11857-1-aneesh.kumar@linux.vnet.ibm.com>
futex_detect_cmpxchg() does a cmpxchg_futex_value_locked on a NULL user addr to
runtime detect whether architecture implements atomic cmpxchg for futex. POWER
do implement the feature and hence we can enable the config instead of depending
on runtime detection.
We could possible enable this on everything. For now limitted to book3s_64
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/platforms/Kconfig.cputype | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index a429d859f15d..31bc2bd5dfd1 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -75,6 +75,7 @@ config PPC_BOOK3S_64
select ARCH_SUPPORTS_NUMA_BALANCING
select IRQ_WORK
select HAVE_KERNEL_XZ
+ select HAVE_FUTEX_CMPXCHG if FUTEX
config PPC_BOOK3E_64
bool "Embedded processors"
--
2.14.3
^ permalink raw reply related
* [RFC PATCH 0/5] Add support for 4PB virtual address space on hash
From: Aneesh Kumar K.V @ 2018-02-10 10:56 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
This patch series extended the max virtual address space value from 512TB
to 4PB with 64K page size. We do that by allocating one vsid context for
each 512TB range. More details of that is explained in patch 4.
Aneesh Kumar K.V (5):
powerpc: Don't do runtime futex_cmpxchg test
powerpc/mm/slice: Update documentation in the file.
powerpc/mm/slice: Reduce the stack usage in slice_get_unmapped_area
powerpc/mm: Add support for handling > 512TB address in SLB miss
powerpc/mm/hash64: Increase the VA range
arch/powerpc/include/asm/book3s/64/hash-4k.h | 6 ++
arch/powerpc/include/asm/book3s/64/hash-64k.h | 7 +-
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 6 +-
arch/powerpc/include/asm/book3s/64/mmu.h | 24 +++++
arch/powerpc/include/asm/processor.h | 9 +-
arch/powerpc/kernel/exceptions-64s.S | 12 ++-
arch/powerpc/mm/copro_fault.c | 2 +-
arch/powerpc/mm/hash_utils_64.c | 4 +-
arch/powerpc/mm/init_64.c | 6 --
arch/powerpc/mm/mmu_context_book3s64.c | 17 ++-
arch/powerpc/mm/pgtable-hash64.c | 2 +-
arch/powerpc/mm/pgtable_64.c | 5 -
arch/powerpc/mm/slb.c | 144 ++++++++++++++++++++++++++
arch/powerpc/mm/slb_low.S | 6 +-
arch/powerpc/mm/slice.c | 61 ++++++-----
arch/powerpc/mm/tlb_hash64.c | 2 +-
arch/powerpc/platforms/Kconfig.cputype | 1 +
17 files changed, 261 insertions(+), 53 deletions(-)
--
2.14.3
^ permalink raw reply
* Re: [PATCH 1/2] powerpc/mm: Fix crashes with PUD level hugetlb config
From: Aneesh Kumar K.V @ 2018-02-10 9:47 UTC (permalink / raw)
To: benh, paulus, mpe, linuxram; +Cc: linuxppc-dev
In-Reply-To: <87r2pvfr5g.fsf@linux.vnet.ibm.com>
Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com> writes:
> "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> writes:
>
>> To support memory keys, we moved the hash pte slot information to the second
>> half of the page table. This was ok with PTE entries at level 4 and level 3.
>> We already allocate larger page table pages at those level to accomodate extra
>> details. For level 4 we already have the extra space which was used to track
>> 4k hash page table entry details and at pmd level the extra space was allocated
>> to track the THP details.
>>
>> With hugetlbfs PTE, we used this extra space at the PMD level to store the
>> slot details. But we also support hugetlbfs PTE at PUD leve and PUD level page
>> didn't allocate extra space. This resulted in memory corruption.
>>
>> Fix this by allocating extra space at PUD level when HUGETLB is enabled. We
>> may need further changes to allocate larger space at PMD level when we enable
>> HUGETLB. That will be done in next patch.
>>
>> Fixes:bf9a95f9a6481bc6e(" powerpc: Free up four 64K PTE bits in 64K backed HPTE pages")
>>
>> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
>
> Another fix, I still get random memory corruption with hugetlb test with
> 16G hugepage config.
Another one. I am not sure whether we really want this in this form. But
with this tests are running fine.
-aneesh
commit 658fe8c310a913e69e5bc9a40d4c28a3b88d5c08
Author: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Date: Sat Feb 10 13:17:34 2018 +0530
powerpc/mm/hash64: memset the pagetable pages on allocation.
Now that we are using second half of the table to store slot details and we
don't clear them in the huge_pte_get_and_clear, we need to make sure we zero
out the range on allocation. This done some extra work because the first half
of the table is cleared by huge_pte_get_and_clear and memset in this patch
zero-out the full table page.
We need to do this for pgd and pud because both get allocated from the same slab
cache.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
The other option is to get huget_pte_get_and_clear to clear the second half of the page table.
That requires generic changes, because we don't have hugetlb page size available there.
diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
index 53df86d3cfce..adb7fba4b6c7 100644
--- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
@@ -73,10 +73,13 @@ static inline void radix__pgd_free(struct mm_struct *mm, pgd_t *pgd)
static inline pgd_t *pgd_alloc(struct mm_struct *mm)
{
+ pgd_t *pgd;
if (radix_enabled())
return radix__pgd_alloc(mm);
- return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE),
- pgtable_gfp_flags(mm, GFP_KERNEL));
+ pgd = kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
+ memset(pgd, 0, PGD_TABLE_SIZE);
+ return pgd;
}
static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
@@ -93,8 +96,11 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)
{
- return kmem_cache_alloc(PGT_CACHE(PUD_CACHE_INDEX),
- pgtable_gfp_flags(mm, GFP_KERNEL));
+ pud_t *pud;
+ pud = kmem_cache_alloc(PGT_CACHE(PUD_CACHE_INDEX),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
+ memset(pud, 0, PUD_TABLE_SIZE);
+ return pud;
}
static inline void pud_free(struct mm_struct *mm, pud_t *pud)
^ permalink raw reply related
* [RFC PATCH 5/5] powerpc/mm/slice: use the dynamic high slice size to limit bitmap operations
From: Nicholas Piggin @ 2018-02-10 8:11 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180210081139.27236-1-npiggin@gmail.com>
The number of high slices a process might use now depends on its
address space size, and what allocation address it has requested.
This patch uses that limit throughout call chains where possible,
rather than use the fixed SLICE_NUM_HIGH for bitmap operations.
This saves some cost for processes that don't use very large address
spaces.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 98 +++++++++++++++++++++++++++----------------------
1 file changed, 55 insertions(+), 43 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index b2e6c7667bc5..bec68ea07e29 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -61,13 +61,12 @@ static void slice_print_mask(const char *label, const struct slice_mask *mask) {
#endif
static void slice_range_to_mask(unsigned long start, unsigned long len,
- struct slice_mask *ret)
+ struct slice_mask *ret,
+ unsigned long high_slices)
{
unsigned long end = start + len - 1;
ret->low_slices = 0;
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
-
if (start < SLICE_LOW_TOP) {
unsigned long mend = min(end, (SLICE_LOW_TOP - 1));
@@ -75,6 +74,7 @@ static void slice_range_to_mask(unsigned long start, unsigned long len,
- (1u << GET_LOW_SLICE_INDEX(start));
}
+ bitmap_zero(ret->high_slices, high_slices);
if ((start + len) > SLICE_LOW_TOP) {
unsigned long start_index = GET_HIGH_SLICE_INDEX(start);
unsigned long align_end = ALIGN(end, (1UL << SLICE_HIGH_SHIFT));
@@ -116,28 +116,27 @@ static int slice_high_has_vma(struct mm_struct *mm, unsigned long slice)
}
static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
- unsigned long high_limit)
+ unsigned long high_slices)
{
unsigned long i;
ret->low_slices = 0;
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
-
for (i = 0; i < SLICE_NUM_LOW; i++)
if (!slice_low_has_vma(mm, i))
ret->low_slices |= 1u << i;
- if (high_limit <= SLICE_LOW_TOP)
+ if (!high_slices)
return;
- for (i = 0; i < GET_HIGH_SLICE_INDEX(high_limit); i++)
+ bitmap_zero(ret->high_slices, high_slices);
+ for (i = 0; i < high_slices; i++)
if (!slice_high_has_vma(mm, i))
__set_bit(i, ret->high_slices);
}
static void calc_slice_mask_for_size(struct mm_struct *mm, int psize,
struct slice_mask *ret,
- unsigned long high_limit)
+ unsigned long high_slices)
{
unsigned char *hpsizes;
int index, mask_index;
@@ -145,18 +144,17 @@ static void calc_slice_mask_for_size(struct mm_struct *mm, int psize,
u64 lpsizes;
ret->low_slices = 0;
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
-
lpsizes = mm->context.low_slices_psize;
for (i = 0; i < SLICE_NUM_LOW; i++)
if (((lpsizes >> (i * 4)) & 0xf) == psize)
ret->low_slices |= 1u << i;
- if (high_limit <= SLICE_LOW_TOP)
+ if (!high_slices)
return;
+ bitmap_zero(ret->high_slices, high_slices);
hpsizes = mm->context.high_slices_psize;
- for (i = 0; i < GET_HIGH_SLICE_INDEX(high_limit); i++) {
+ for (i = 0; i < high_slices; i++) {
mask_index = i & 0x1;
index = i >> 1;
if (((hpsizes[index] >> (mask_index * 4)) & 0xf) == psize)
@@ -165,16 +163,15 @@ static void calc_slice_mask_for_size(struct mm_struct *mm, int psize,
}
#ifdef CONFIG_PPC_BOOK3S_64
-static void recalc_slice_mask_cache(struct mm_struct *mm)
+static void recalc_slice_mask_cache(struct mm_struct *mm, unsigned long high_slices)
{
- unsigned long l = mm->context.slb_addr_limit;
- calc_slice_mask_for_size(mm, MMU_PAGE_4K, &mm->context.mask_4k, l);
+ calc_slice_mask_for_size(mm, MMU_PAGE_4K, &mm->context.mask_4k, high_slices);
#ifdef CONFIG_PPC_64K_PAGES
- calc_slice_mask_for_size(mm, MMU_PAGE_64K, &mm->context.mask_64k, l);
+ calc_slice_mask_for_size(mm, MMU_PAGE_64K, &mm->context.mask_64k, high_slices);
#endif
#ifdef CONFIG_HUGETLB_PAGE
- calc_slice_mask_for_size(mm, MMU_PAGE_16M, &mm->context.mask_16m, l);
- calc_slice_mask_for_size(mm, MMU_PAGE_16G, &mm->context.mask_16g, l);
+ calc_slice_mask_for_size(mm, MMU_PAGE_16M, &mm->context.mask_16m, high_slices);
+ calc_slice_mask_for_size(mm, MMU_PAGE_16G, &mm->context.mask_16g, high_slices);
#endif
}
@@ -252,6 +249,7 @@ static void slice_convert(struct mm_struct *mm,
unsigned char *hpsizes;
u64 lpsizes;
unsigned long i, flags;
+ unsigned long high_slices;
slice_dbg("slice_convert(mm=%p, psize=%d)\n", mm, psize);
slice_print_mask(" mask", mask);
@@ -271,7 +269,8 @@ static void slice_convert(struct mm_struct *mm,
mm->context.low_slices_psize = lpsizes;
hpsizes = mm->context.high_slices_psize;
- for (i = 0; i < GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit); i++) {
+ high_slices = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ for (i = 0; i < high_slices; i++) {
mask_index = i & 0x1;
index = i >> 1;
if (test_bit(i, mask->high_slices))
@@ -284,7 +283,7 @@ static void slice_convert(struct mm_struct *mm,
(unsigned long)mm->context.low_slices_psize,
(unsigned long)mm->context.high_slices_psize);
- recalc_slice_mask_cache(mm);
+ recalc_slice_mask_cache(mm, high_slices);
spin_unlock_irqrestore(&slice_convert_lock, flags);
@@ -431,27 +430,32 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
}
static inline void slice_copy_mask(struct slice_mask *dst,
- const struct slice_mask *src)
+ const struct slice_mask *src,
+ unsigned long high_slices)
{
dst->low_slices = src->low_slices;
- bitmap_copy(dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
+ bitmap_copy(dst->high_slices, src->high_slices, high_slices);
}
static inline void slice_or_mask(struct slice_mask *dst,
const struct slice_mask *src1,
- const struct slice_mask *src2)
+ const struct slice_mask *src2,
+ unsigned long high_slices)
{
dst->low_slices = src1->low_slices | src2->low_slices;
- bitmap_or(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
+ bitmap_or(dst->high_slices, src1->high_slices, src2->high_slices,
+ high_slices);
}
static inline void slice_andnot_mask(struct slice_mask *dst,
const struct slice_mask *src1,
- const struct slice_mask *src2)
+ const struct slice_mask *src2,
+ unsigned long high_slices)
{
dst->low_slices = src1->low_slices & ~src2->low_slices;
- bitmap_andnot(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
+ bitmap_andnot(dst->high_slices, src1->high_slices, src2->high_slices,
+ high_slices);
}
#ifdef CONFIG_PPC_64K_PAGES
@@ -474,6 +478,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
struct mm_struct *mm = current->mm;
unsigned long newaddr;
unsigned long high_limit;
+ unsigned long high_slices;
high_limit = DEFAULT_MAP_WINDOW;
if (addr >= high_limit || (fixed && (addr + len > high_limit)))
@@ -490,13 +495,14 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
return -ENOMEM;
}
+ high_slices = GET_HIGH_SLICE_INDEX(high_limit);
if (high_limit > mm->context.slb_addr_limit) {
unsigned long flags;
mm->context.slb_addr_limit = high_limit;
spin_lock_irqsave(&slice_convert_lock, flags);
- recalc_slice_mask_cache(mm);
+ recalc_slice_mask_cache(mm, high_slices);
spin_unlock_irqrestore(&slice_convert_lock, flags);
on_each_cpu(slice_flush_segments, mm, 1);
@@ -504,7 +510,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* silence stupid warning */;
potential_mask.low_slices = 0;
- bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
+ bitmap_zero(potential_mask.high_slices, high_slices);
/* Sanity checks */
BUG_ON(mm->task_size == 0);
@@ -555,13 +561,13 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (psize == MMU_PAGE_64K) {
compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
if (fixed)
- slice_or_mask(&good_mask, maskp, compat_maskp);
+ slice_or_mask(&good_mask, maskp, compat_maskp, high_slices);
else
- slice_copy_mask(&good_mask, maskp);
+ slice_copy_mask(&good_mask, maskp, high_slices);
} else
#endif
{
- slice_copy_mask(&good_mask, maskp);
+ slice_copy_mask(&good_mask, maskp, high_slices);
}
/* First check hint if it's valid or if we have MAP_FIXED */
@@ -591,8 +597,8 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* We don't fit in the good mask, check what other slices are
* empty and thus can be converted
*/
- slice_mask_for_free(mm, &potential_mask, high_limit);
- slice_or_mask(&potential_mask, &potential_mask, &good_mask);
+ slice_mask_for_free(mm, &potential_mask, high_slices);
+ slice_or_mask(&potential_mask, &potential_mask, &good_mask, high_slices);
slice_print_mask(" potential", &potential_mask);
if (addr || fixed) {
@@ -629,7 +635,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
if (addr == -ENOMEM && psize == MMU_PAGE_64K) {
/* retry the search with 4k-page slices included */
- slice_or_mask(&potential_mask, &potential_mask, compat_maskp);
+ slice_or_mask(&potential_mask, &potential_mask, compat_maskp, high_slices);
addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
}
@@ -638,16 +644,16 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (addr == -ENOMEM)
return -ENOMEM;
- slice_range_to_mask(addr, len, &potential_mask);
+ slice_range_to_mask(addr, len, &potential_mask, high_slices);
slice_dbg(" found potential area at 0x%lx\n", addr);
slice_print_mask(" mask", maskp);
convert:
- slice_andnot_mask(&potential_mask, &potential_mask, &good_mask);
+ slice_andnot_mask(&potential_mask, &potential_mask, &good_mask, high_slices);
if (compat_maskp && !fixed)
- slice_andnot_mask(&potential_mask, &potential_mask, compat_maskp);
+ slice_andnot_mask(&potential_mask, &potential_mask, compat_maskp, high_slices);
if (potential_mask.low_slices ||
- !bitmap_empty(potential_mask.high_slices, SLICE_NUM_HIGH)) {
+ !bitmap_empty(potential_mask.high_slices, high_slices)) {
slice_convert(mm, &potential_mask, psize);
if (psize > MMU_PAGE_BASE)
on_each_cpu(slice_flush_segments, mm, 1);
@@ -724,6 +730,7 @@ void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
int index, mask_index;
unsigned char *hpsizes;
unsigned long flags, lpsizes;
+ unsigned long high_slices;
unsigned int old_psize;
int i;
@@ -749,7 +756,8 @@ void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
mm->context.low_slices_psize = lpsizes;
hpsizes = mm->context.high_slices_psize;
- for (i = 0; i < SLICE_NUM_HIGH; i++) {
+ high_slices = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ for (i = 0; i < high_slices; i++) {
mask_index = i & 0x1;
index = i >> 1;
if (((hpsizes[index] >> (mask_index * 4)) & 0xf) == old_psize)
@@ -765,7 +773,7 @@ void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
(unsigned long)mm->context.low_slices_psize,
(unsigned long)mm->context.high_slices_psize);
- recalc_slice_mask_cache(mm);
+ recalc_slice_mask_cache(mm, high_slices);
spin_unlock_irqrestore(&slice_convert_lock, flags);
return;
bail:
@@ -776,10 +784,12 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
unsigned long len, unsigned int psize)
{
struct slice_mask mask;
+ unsigned long high_slices;
VM_BUG_ON(radix_enabled());
- slice_range_to_mask(start, len, &mask);
+ high_slices = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ slice_range_to_mask(start, len, &mask, high_slices);
slice_convert(mm, &mask, psize);
}
@@ -818,9 +828,11 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
if (psize == MMU_PAGE_64K) {
const struct slice_mask *compat_maskp;
struct slice_mask available;
+ unsigned long high_slices;
compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
- slice_or_mask(&available, maskp, compat_maskp);
+ high_slices = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ slice_or_mask(&available, maskp, compat_maskp, high_slices);
return !slice_check_range_fits(mm, &available, addr, len);
}
#endif
--
2.15.1
^ permalink raw reply related
* [RFC PATCH 4/5] powerpc/mm/slice: Use const pointers to cached slice masks where possible
From: Nicholas Piggin @ 2018-02-10 8:11 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180210081139.27236-1-npiggin@gmail.com>
The slice_mask cache was a basic conversion which copied the slice
mask into caller's structures, because that's how the original code
worked. In most cases the pointer can be used directly instead, saving
a copy and an on-stack structure.
This also converts the slice_mask bit operation helpers to be the usual
3-operand kind, which is clearer to work with. And we remove some
unnecessary intermediate bitmaps, reducing stack and copy overhead
further.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 78 ++++++++++++++++++++++++++++---------------------
1 file changed, 44 insertions(+), 34 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 98497c105d7d..b2e6c7667bc5 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -430,25 +430,28 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
return slice_find_area_bottomup(mm, len, mask, psize, high_limit);
}
-static inline void slice_or_mask(struct slice_mask *dst,
+static inline void slice_copy_mask(struct slice_mask *dst,
const struct slice_mask *src)
{
- DECLARE_BITMAP(result, SLICE_NUM_HIGH);
+ dst->low_slices = src->low_slices;
+ bitmap_copy(dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
+}
- dst->low_slices |= src->low_slices;
- bitmap_or(result, dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
- bitmap_copy(dst->high_slices, result, SLICE_NUM_HIGH);
+static inline void slice_or_mask(struct slice_mask *dst,
+ const struct slice_mask *src1,
+ const struct slice_mask *src2)
+{
+ dst->low_slices = src1->low_slices | src2->low_slices;
+ bitmap_or(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
}
static inline void slice_andnot_mask(struct slice_mask *dst,
- const struct slice_mask *src)
+ const struct slice_mask *src1,
+ const struct slice_mask *src2)
{
- DECLARE_BITMAP(result, SLICE_NUM_HIGH);
-
- dst->low_slices &= ~src->low_slices;
+ dst->low_slices = src1->low_slices & ~src2->low_slices;
- bitmap_andnot(result, dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
- bitmap_copy(dst->high_slices, result, SLICE_NUM_HIGH);
+ bitmap_andnot(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
}
#ifdef CONFIG_PPC_64K_PAGES
@@ -461,10 +464,10 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
unsigned long flags, unsigned int psize,
int topdown)
{
- struct slice_mask mask;
struct slice_mask good_mask;
struct slice_mask potential_mask;
- struct slice_mask compat_mask;
+ const struct slice_mask *maskp;
+ const struct slice_mask *compat_maskp = NULL;
int fixed = (flags & MAP_FIXED);
int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT);
unsigned long page_size = 1UL << pshift;
@@ -503,9 +506,6 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
potential_mask.low_slices = 0;
bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
- compat_mask.low_slices = 0;
- bitmap_zero(compat_mask.high_slices, SLICE_NUM_HIGH);
-
/* Sanity checks */
BUG_ON(mm->task_size == 0);
BUG_ON(mm->context.slb_addr_limit == 0);
@@ -528,7 +528,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* First make up a "good" mask of slices that have the right size
* already
*/
- good_mask = *slice_mask_for_size(mm, psize);
+ maskp = slice_mask_for_size(mm, psize);
slice_print_mask(" good_mask", &good_mask);
/*
@@ -553,11 +553,16 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
/* If we support combo pages, we can allow 64k pages in 4k slices */
if (psize == MMU_PAGE_64K) {
- compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
+ compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
if (fixed)
- slice_or_mask(&good_mask, &compat_mask);
- }
+ slice_or_mask(&good_mask, maskp, compat_maskp);
+ else
+ slice_copy_mask(&good_mask, maskp);
+ } else
#endif
+ {
+ slice_copy_mask(&good_mask, maskp);
+ }
/* First check hint if it's valid or if we have MAP_FIXED */
if (addr || fixed) {
@@ -587,7 +592,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* empty and thus can be converted
*/
slice_mask_for_free(mm, &potential_mask, high_limit);
- slice_or_mask(&potential_mask, &good_mask);
+ slice_or_mask(&potential_mask, &potential_mask, &good_mask);
slice_print_mask(" potential", &potential_mask);
if (addr || fixed) {
@@ -624,7 +629,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
if (addr == -ENOMEM && psize == MMU_PAGE_64K) {
/* retry the search with 4k-page slices included */
- slice_or_mask(&potential_mask, &compat_mask);
+ slice_or_mask(&potential_mask, &potential_mask, compat_maskp);
addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
}
@@ -633,15 +638,17 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (addr == -ENOMEM)
return -ENOMEM;
- slice_range_to_mask(addr, len, &mask);
+ slice_range_to_mask(addr, len, &potential_mask);
slice_dbg(" found potential area at 0x%lx\n", addr);
- slice_print_mask(" mask", &mask);
+ slice_print_mask(" mask", maskp);
convert:
- slice_andnot_mask(&mask, &good_mask);
- slice_andnot_mask(&mask, &compat_mask);
- if (mask.low_slices || !bitmap_empty(mask.high_slices, SLICE_NUM_HIGH)) {
- slice_convert(mm, &mask, psize);
+ slice_andnot_mask(&potential_mask, &potential_mask, &good_mask);
+ if (compat_maskp && !fixed)
+ slice_andnot_mask(&potential_mask, &potential_mask, compat_maskp);
+ if (potential_mask.low_slices ||
+ !bitmap_empty(potential_mask.high_slices, SLICE_NUM_HIGH)) {
+ slice_convert(mm, &potential_mask, psize);
if (psize > MMU_PAGE_BASE)
on_each_cpu(slice_flush_segments, mm, 1);
}
@@ -799,19 +806,22 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
unsigned long len)
{
- struct slice_mask available;
+ const struct slice_mask *maskp;
unsigned int psize = mm->context.user_psize;
if (radix_enabled())
return 0;
- available = *slice_mask_for_size(mm, psize);
+ maskp = slice_mask_for_size(mm, psize);
#ifdef CONFIG_PPC_64K_PAGES
/* We need to account for 4k slices too */
if (psize == MMU_PAGE_64K) {
- struct slice_mask compat_mask;
- compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
- slice_or_mask(&available, &compat_mask);
+ const struct slice_mask *compat_maskp;
+ struct slice_mask available;
+
+ compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
+ slice_or_mask(&available, maskp, compat_maskp);
+ return !slice_check_range_fits(mm, &available, addr, len);
}
#endif
@@ -821,6 +831,6 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
slice_print_mask(" mask", &mask);
slice_print_mask(" available", &available);
#endif
- return !slice_check_range_fits(mm, &available, addr, len);
+ return !slice_check_range_fits(mm, maskp, addr, len);
}
#endif
--
2.15.1
^ permalink raw reply related
* [RFC PATCH 3/5] powerpc/mm/slice: implement slice_check_range_fits
From: Nicholas Piggin @ 2018-02-10 8:11 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180210081139.27236-1-npiggin@gmail.com>
Rather than build slice masks from a range then use that to check for
fit in a candidate mask, implement slice_check_range_fits that checks
if a range fits in a mask directly.
This allows several structures to be removed from stacks, and also we
don't expect a huge range in a lot of these cases, so building and
comparing a full mask is going to be more expensive than testing just
one or two bits of the range.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 67 ++++++++++++++++++++++++++-----------------------
1 file changed, 35 insertions(+), 32 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 837700bb50a9..98497c105d7d 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -198,22 +198,35 @@ static const struct slice_mask *slice_mask_for_size(struct mm_struct *mm, int ps
#error "Must define the slice masks for page sizes supported by the platform"
#endif
-static int slice_check_fit(struct mm_struct *mm,
- const struct slice_mask *mask,
- const struct slice_mask *available)
+static bool slice_check_range_fits(struct mm_struct *mm,
+ const struct slice_mask *available,
+ unsigned long start, unsigned long len)
{
- DECLARE_BITMAP(result, SLICE_NUM_HIGH);
- /*
- * Make sure we just do bit compare only to the max
- * addr limit and not the full bit map size.
- */
- unsigned long slice_count = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ unsigned long end = start + len - 1;
+ u64 low_slices = 0;
+
+ if (start < SLICE_LOW_TOP) {
+ unsigned long mend = min(end, (SLICE_LOW_TOP - 1));
+
+ low_slices = (1u << (GET_LOW_SLICE_INDEX(mend) + 1))
+ - (1u << GET_LOW_SLICE_INDEX(start));
+ }
+ if ((low_slices & available->low_slices) != low_slices)
+ return false;
+
+ if ((start + len) > SLICE_LOW_TOP) {
+ unsigned long start_index = GET_HIGH_SLICE_INDEX(start);
+ unsigned long align_end = ALIGN(end, (1UL << SLICE_HIGH_SHIFT));
+ unsigned long count = GET_HIGH_SLICE_INDEX(align_end) - start_index;
+ unsigned long i;
- bitmap_and(result, mask->high_slices,
- available->high_slices, slice_count);
+ for (i = start_index; i < start_index + count; i++) {
+ if (!test_bit(i, available->high_slices))
+ return false;
+ }
+ }
- return (mask->low_slices & available->low_slices) == mask->low_slices &&
- bitmap_equal(result, mask->high_slices, slice_count);
+ return true;
}
static void slice_flush_segments(void *parm)
@@ -486,12 +499,6 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
on_each_cpu(slice_flush_segments, mm, 1);
}
- /*
- * init different masks
- */
- mask.low_slices = 0;
- bitmap_zero(mask.high_slices, SLICE_NUM_HIGH);
-
/* silence stupid warning */;
potential_mask.low_slices = 0;
bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
@@ -553,15 +560,11 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#endif
/* First check hint if it's valid or if we have MAP_FIXED */
- if (addr != 0 || fixed) {
- /* Build a mask for the requested range */
- slice_range_to_mask(addr, len, &mask);
- slice_print_mask(" mask", &mask);
-
+ if (addr || fixed) {
/* Check if we fit in the good mask. If we do, we just return,
* nothing else to do
*/
- if (slice_check_fit(mm, &mask, &good_mask)) {
+ if (slice_check_range_fits(mm, &good_mask, addr, len)) {
slice_dbg(" fits good !\n");
return addr;
}
@@ -587,10 +590,11 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
slice_or_mask(&potential_mask, &good_mask);
slice_print_mask(" potential", &potential_mask);
- if ((addr != 0 || fixed) &&
- slice_check_fit(mm, &mask, &potential_mask)) {
- slice_dbg(" fits potential !\n");
- goto convert;
+ if (addr || fixed) {
+ if (slice_check_range_fits(mm, &potential_mask, addr, len)) {
+ slice_dbg(" fits potential !\n");
+ goto convert;
+ }
}
/* If we have MAP_FIXED and failed the above steps, then error out */
@@ -795,13 +799,12 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
unsigned long len)
{
- struct slice_mask mask, available;
+ struct slice_mask available;
unsigned int psize = mm->context.user_psize;
if (radix_enabled())
return 0;
- slice_range_to_mask(addr, len, &mask);
available = *slice_mask_for_size(mm, psize);
#ifdef CONFIG_PPC_64K_PAGES
/* We need to account for 4k slices too */
@@ -818,6 +821,6 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
slice_print_mask(" mask", &mask);
slice_print_mask(" available", &available);
#endif
- return !slice_check_fit(mm, &mask, &available);
+ return !slice_check_range_fits(mm, &available, addr, len);
}
#endif
--
2.15.1
^ permalink raw reply related
* [RFC PATCH 2/5] powerpc/mm/slice: implement a slice mask cache
From: Nicholas Piggin @ 2018-02-10 8:11 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180210081139.27236-1-npiggin@gmail.com>
Calculating the slice mask can become a signifcant overhead for
get_unmapped_area. This patch adds a struct slice_mask for
each page size in the mm_context, and keeps these in synch with
the slices psize arrays and slb_addr_limit.
This saves about 30% kernel time on a single-page mmap/munmap micro
benchmark.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/book3s/64/mmu.h | 20 +++++++++-
arch/powerpc/mm/slice.c | 68 ++++++++++++++++++++++++--------
2 files changed, 71 insertions(+), 17 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
index 0abeb0e2d616..b6d136fd8ffd 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu.h
@@ -80,6 +80,16 @@ struct spinlock;
/* Maximum possible number of NPUs in a system. */
#define NV_MAX_NPUS 8
+/*
+ * One bit per slice. We have lower slices which cover 256MB segments
+ * upto 4G range. That gets us 16 low slices. For the rest we track slices
+ * in 1TB size.
+ */
+struct slice_mask {
+ u64 low_slices;
+ DECLARE_BITMAP(high_slices, SLICE_NUM_HIGH);
+};
+
typedef struct {
mm_context_id_t id;
u16 user_psize; /* page size index */
@@ -91,9 +101,17 @@ typedef struct {
struct npu_context *npu_context;
#ifdef CONFIG_PPC_MM_SLICES
+ unsigned long slb_addr_limit;
u64 low_slices_psize; /* SLB page size encodings */
unsigned char high_slices_psize[SLICE_ARRAY_SIZE];
- unsigned long slb_addr_limit;
+# ifdef CONFIG_PPC_64K_PAGES
+ struct slice_mask mask_64k;
+# endif
+ struct slice_mask mask_4k;
+# ifdef CONFIG_HUGETLB_PAGE
+ struct slice_mask mask_16m;
+ struct slice_mask mask_16g;
+# endif
#else
u16 sllp; /* SLB page size encoding */
#endif
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index e8f6922d3c9b..837700bb50a9 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -37,15 +37,6 @@
#include <asm/hugetlb.h>
static DEFINE_SPINLOCK(slice_convert_lock);
-/*
- * One bit per slice. We have lower slices which cover 256MB segments
- * upto 4G range. That gets us 16 low slices. For the rest we track slices
- * in 1TB size.
- */
-struct slice_mask {
- u64 low_slices;
- DECLARE_BITMAP(high_slices, SLICE_NUM_HIGH);
-};
#ifdef DEBUG
int _slice_debug = 1;
@@ -144,7 +135,7 @@ static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
__set_bit(i, ret->high_slices);
}
-static void slice_mask_for_size(struct mm_struct *mm, int psize,
+static void calc_slice_mask_for_size(struct mm_struct *mm, int psize,
struct slice_mask *ret,
unsigned long high_limit)
{
@@ -173,6 +164,40 @@ static void slice_mask_for_size(struct mm_struct *mm, int psize,
}
}
+#ifdef CONFIG_PPC_BOOK3S_64
+static void recalc_slice_mask_cache(struct mm_struct *mm)
+{
+ unsigned long l = mm->context.slb_addr_limit;
+ calc_slice_mask_for_size(mm, MMU_PAGE_4K, &mm->context.mask_4k, l);
+#ifdef CONFIG_PPC_64K_PAGES
+ calc_slice_mask_for_size(mm, MMU_PAGE_64K, &mm->context.mask_64k, l);
+#endif
+#ifdef CONFIG_HUGETLB_PAGE
+ calc_slice_mask_for_size(mm, MMU_PAGE_16M, &mm->context.mask_16m, l);
+ calc_slice_mask_for_size(mm, MMU_PAGE_16G, &mm->context.mask_16g, l);
+#endif
+}
+
+static const struct slice_mask *slice_mask_for_size(struct mm_struct *mm, int psize)
+{
+#ifdef CONFIG_PPC_64K_PAGES
+ if (psize == MMU_PAGE_64K)
+ return &mm->context.mask_64k;
+#endif
+ if (psize == MMU_PAGE_4K)
+ return &mm->context.mask_4k;
+#ifdef CONFIG_HUGETLB_PAGE
+ if (psize == MMU_PAGE_16M)
+ return &mm->context.mask_16m;
+ if (psize == MMU_PAGE_16G)
+ return &mm->context.mask_16g;
+#endif
+ BUG();
+}
+#else
+#error "Must define the slice masks for page sizes supported by the platform"
+#endif
+
static int slice_check_fit(struct mm_struct *mm,
const struct slice_mask *mask,
const struct slice_mask *available)
@@ -246,6 +271,8 @@ static void slice_convert(struct mm_struct *mm,
(unsigned long)mm->context.low_slices_psize,
(unsigned long)mm->context.high_slices_psize);
+ recalc_slice_mask_cache(mm);
+
spin_unlock_irqrestore(&slice_convert_lock, flags);
copro_flush_all_slbs(mm);
@@ -448,7 +475,14 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
}
if (high_limit > mm->context.slb_addr_limit) {
+ unsigned long flags;
+
mm->context.slb_addr_limit = high_limit;
+
+ spin_lock_irqsave(&slice_convert_lock, flags);
+ recalc_slice_mask_cache(mm);
+ spin_unlock_irqrestore(&slice_convert_lock, flags);
+
on_each_cpu(slice_flush_segments, mm, 1);
}
@@ -487,7 +521,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* First make up a "good" mask of slices that have the right size
* already
*/
- slice_mask_for_size(mm, psize, &good_mask, high_limit);
+ good_mask = *slice_mask_for_size(mm, psize);
slice_print_mask(" good_mask", &good_mask);
/*
@@ -512,7 +546,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
/* If we support combo pages, we can allow 64k pages in 4k slices */
if (psize == MMU_PAGE_64K) {
- slice_mask_for_size(mm, MMU_PAGE_4K, &compat_mask, high_limit);
+ compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
if (fixed)
slice_or_mask(&good_mask, &compat_mask);
}
@@ -693,7 +727,7 @@ void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
goto bail;
mm->context.user_psize = psize;
- wmb();
+ wmb(); /* Why? */
lpsizes = mm->context.low_slices_psize;
for (i = 0; i < SLICE_NUM_LOW; i++)
@@ -720,6 +754,9 @@ void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
(unsigned long)mm->context.low_slices_psize,
(unsigned long)mm->context.high_slices_psize);
+ recalc_slice_mask_cache(mm);
+ spin_unlock_irqrestore(&slice_convert_lock, flags);
+ return;
bail:
spin_unlock_irqrestore(&slice_convert_lock, flags);
}
@@ -760,18 +797,17 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
{
struct slice_mask mask, available;
unsigned int psize = mm->context.user_psize;
- unsigned long high_limit = mm->context.slb_addr_limit;
if (radix_enabled())
return 0;
slice_range_to_mask(addr, len, &mask);
- slice_mask_for_size(mm, psize, &available, high_limit);
+ available = *slice_mask_for_size(mm, psize);
#ifdef CONFIG_PPC_64K_PAGES
/* We need to account for 4k slices too */
if (psize == MMU_PAGE_64K) {
struct slice_mask compat_mask;
- slice_mask_for_size(mm, MMU_PAGE_4K, &compat_mask, high_limit);
+ compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
slice_or_mask(&available, &compat_mask);
}
#endif
--
2.15.1
^ permalink raw reply related
* [RFC PATCH 1/5] powerpc/mm/slice: pass pointers to struct slice_mask where possible
From: Nicholas Piggin @ 2018-02-10 8:11 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180210081139.27236-1-npiggin@gmail.com>
Pass around const pointers to struct slice_mask where possible, rather
than copies of slice_mask, to reduce stack and call overhead.
checkstack.pl gives, before:
0x00000de4 slice_get_unmapped_area [slice.o]: 656
0x00001b4c is_hugepage_only_range [slice.o]: 512
0x0000075c slice_find_area_topdown [slice.o]: 416
0x000004c8 slice_find_area_bottomup.isra.1 [slice.o]: 272
0x00001aa0 slice_set_range_psize [slice.o]: 240
0x00000a64 slice_find_area [slice.o]: 176
0x00000174 slice_check_fit [slice.o]: 112
after:
0x00000bd4 slice_get_unmapped_area [slice.o]: 496
0x000017cc is_hugepage_only_range [slice.o]: 352
0x00000758 slice_find_area [slice.o]: 144
0x00001750 slice_set_range_psize [slice.o]: 144
0x00000180 slice_check_fit [slice.o]: 128
0x000005b0 slice_find_area_bottomup.isra.2 [slice.o]: 128
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 83 +++++++++++++++++++++++++++----------------------
1 file changed, 45 insertions(+), 38 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 23ec2c5e3b78..e8f6922d3c9b 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -50,19 +50,21 @@ struct slice_mask {
#ifdef DEBUG
int _slice_debug = 1;
-static void slice_print_mask(const char *label, struct slice_mask mask)
+static void slice_print_mask(const char *label, const struct slice_mask *mask)
{
if (!_slice_debug)
return;
- pr_devel("%s low_slice: %*pbl\n", label, (int)SLICE_NUM_LOW, &mask.low_slices);
- pr_devel("%s high_slice: %*pbl\n", label, (int)SLICE_NUM_HIGH, mask.high_slices);
+ pr_devel("%s low_slice: %*pbl\n", label,
+ (int)SLICE_NUM_LOW, &mask->low_slices);
+ pr_devel("%s high_slice: %*pbl\n", label,
+ (int)SLICE_NUM_HIGH, mask->high_slices);
}
#define slice_dbg(fmt...) do { if (_slice_debug) pr_devel(fmt); } while (0)
#else
-static void slice_print_mask(const char *label, struct slice_mask mask) {}
+static void slice_print_mask(const char *label, const struct slice_mask *mask) {}
#define slice_dbg(fmt...)
#endif
@@ -142,7 +144,8 @@ static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
__set_bit(i, ret->high_slices);
}
-static void slice_mask_for_size(struct mm_struct *mm, int psize, struct slice_mask *ret,
+static void slice_mask_for_size(struct mm_struct *mm, int psize,
+ struct slice_mask *ret,
unsigned long high_limit)
{
unsigned char *hpsizes;
@@ -171,7 +174,8 @@ static void slice_mask_for_size(struct mm_struct *mm, int psize, struct slice_ma
}
static int slice_check_fit(struct mm_struct *mm,
- struct slice_mask mask, struct slice_mask available)
+ const struct slice_mask *mask,
+ const struct slice_mask *available)
{
DECLARE_BITMAP(result, SLICE_NUM_HIGH);
/*
@@ -180,11 +184,11 @@ static int slice_check_fit(struct mm_struct *mm,
*/
unsigned long slice_count = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
- bitmap_and(result, mask.high_slices,
- available.high_slices, slice_count);
+ bitmap_and(result, mask->high_slices,
+ available->high_slices, slice_count);
- return (mask.low_slices & available.low_slices) == mask.low_slices &&
- bitmap_equal(result, mask.high_slices, slice_count);
+ return (mask->low_slices & available->low_slices) == mask->low_slices &&
+ bitmap_equal(result, mask->high_slices, slice_count);
}
static void slice_flush_segments(void *parm)
@@ -202,7 +206,8 @@ static void slice_flush_segments(void *parm)
local_irq_restore(flags);
}
-static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psize)
+static void slice_convert(struct mm_struct *mm,
+ const struct slice_mask *mask, int psize)
{
int index, mask_index;
/* Write the new slice psize bits */
@@ -220,7 +225,7 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
lpsizes = mm->context.low_slices_psize;
for (i = 0; i < SLICE_NUM_LOW; i++)
- if (mask.low_slices & (1u << i))
+ if (mask->low_slices & (1u << i))
lpsizes = (lpsizes & ~(0xful << (i * 4))) |
(((unsigned long)psize) << (i * 4));
@@ -231,7 +236,7 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
for (i = 0; i < GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit); i++) {
mask_index = i & 0x1;
index = i >> 1;
- if (test_bit(i, mask.high_slices))
+ if (test_bit(i, mask->high_slices))
hpsizes[index] = (hpsizes[index] &
~(0xf << (mask_index * 4))) |
(((unsigned long)psize) << (mask_index * 4));
@@ -254,26 +259,25 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
* 'available' slice_mark.
*/
static bool slice_scan_available(unsigned long addr,
- struct slice_mask available,
- int end,
- unsigned long *boundary_addr)
+ const struct slice_mask *available,
+ int end, unsigned long *boundary_addr)
{
unsigned long slice;
if (addr < SLICE_LOW_TOP) {
slice = GET_LOW_SLICE_INDEX(addr);
*boundary_addr = (slice + end) << SLICE_LOW_SHIFT;
- return !!(available.low_slices & (1u << slice));
+ return !!(available->low_slices & (1u << slice));
} else {
slice = GET_HIGH_SLICE_INDEX(addr);
*boundary_addr = (slice + end) ?
((slice + end) << SLICE_HIGH_SHIFT) : SLICE_LOW_TOP;
- return !!test_bit(slice, available.high_slices);
+ return !!test_bit(slice, available->high_slices);
}
}
static unsigned long slice_find_area_bottomup(struct mm_struct *mm,
unsigned long len,
- struct slice_mask available,
+ const struct slice_mask *available,
int psize, unsigned long high_limit)
{
int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT);
@@ -319,7 +323,7 @@ static unsigned long slice_find_area_bottomup(struct mm_struct *mm,
static unsigned long slice_find_area_topdown(struct mm_struct *mm,
unsigned long len,
- struct slice_mask available,
+ const struct slice_mask *available,
int psize, unsigned long high_limit)
{
int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT);
@@ -377,7 +381,7 @@ static unsigned long slice_find_area_topdown(struct mm_struct *mm,
static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
- struct slice_mask mask, int psize,
+ const struct slice_mask *mask, int psize,
int topdown, unsigned long high_limit)
{
if (topdown)
@@ -386,7 +390,8 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
return slice_find_area_bottomup(mm, len, mask, psize, high_limit);
}
-static inline void slice_or_mask(struct slice_mask *dst, struct slice_mask *src)
+static inline void slice_or_mask(struct slice_mask *dst,
+ const struct slice_mask *src)
{
DECLARE_BITMAP(result, SLICE_NUM_HIGH);
@@ -395,7 +400,8 @@ static inline void slice_or_mask(struct slice_mask *dst, struct slice_mask *src)
bitmap_copy(dst->high_slices, result, SLICE_NUM_HIGH);
}
-static inline void slice_andnot_mask(struct slice_mask *dst, struct slice_mask *src)
+static inline void slice_andnot_mask(struct slice_mask *dst,
+ const struct slice_mask *src)
{
DECLARE_BITMAP(result, SLICE_NUM_HIGH);
@@ -482,7 +488,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* already
*/
slice_mask_for_size(mm, psize, &good_mask, high_limit);
- slice_print_mask(" good_mask", good_mask);
+ slice_print_mask(" good_mask", &good_mask);
/*
* Here "good" means slices that are already the right page size,
@@ -516,12 +522,12 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (addr != 0 || fixed) {
/* Build a mask for the requested range */
slice_range_to_mask(addr, len, &mask);
- slice_print_mask(" mask", mask);
+ slice_print_mask(" mask", &mask);
/* Check if we fit in the good mask. If we do, we just return,
* nothing else to do
*/
- if (slice_check_fit(mm, mask, good_mask)) {
+ if (slice_check_fit(mm, &mask, &good_mask)) {
slice_dbg(" fits good !\n");
return addr;
}
@@ -529,7 +535,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* Now let's see if we can find something in the existing
* slices for that size
*/
- newaddr = slice_find_area(mm, len, good_mask,
+ newaddr = slice_find_area(mm, len, &good_mask,
psize, topdown, high_limit);
if (newaddr != -ENOMEM) {
/* Found within the good mask, we don't have to setup,
@@ -545,9 +551,10 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
*/
slice_mask_for_free(mm, &potential_mask, high_limit);
slice_or_mask(&potential_mask, &good_mask);
- slice_print_mask(" potential", potential_mask);
+ slice_print_mask(" potential", &potential_mask);
- if ((addr != 0 || fixed) && slice_check_fit(mm, mask, potential_mask)) {
+ if ((addr != 0 || fixed) &&
+ slice_check_fit(mm, &mask, &potential_mask)) {
slice_dbg(" fits potential !\n");
goto convert;
}
@@ -562,7 +569,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* anywhere in the good area.
*/
if (addr) {
- addr = slice_find_area(mm, len, good_mask,
+ addr = slice_find_area(mm, len, &good_mask,
psize, topdown, high_limit);
if (addr != -ENOMEM) {
slice_dbg(" found area at 0x%lx\n", addr);
@@ -573,14 +580,14 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* Now let's see if we can find something in the existing slices
* for that size plus free slices
*/
- addr = slice_find_area(mm, len, potential_mask,
+ addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
#ifdef CONFIG_PPC_64K_PAGES
if (addr == -ENOMEM && psize == MMU_PAGE_64K) {
/* retry the search with 4k-page slices included */
slice_or_mask(&potential_mask, &compat_mask);
- addr = slice_find_area(mm, len, potential_mask,
+ addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
}
#endif
@@ -590,13 +597,13 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
slice_range_to_mask(addr, len, &mask);
slice_dbg(" found potential area at 0x%lx\n", addr);
- slice_print_mask(" mask", mask);
+ slice_print_mask(" mask", &mask);
convert:
slice_andnot_mask(&mask, &good_mask);
slice_andnot_mask(&mask, &compat_mask);
if (mask.low_slices || !bitmap_empty(mask.high_slices, SLICE_NUM_HIGH)) {
- slice_convert(mm, mask, psize);
+ slice_convert(mm, &mask, psize);
if (psize > MMU_PAGE_BASE)
on_each_cpu(slice_flush_segments, mm, 1);
}
@@ -725,7 +732,7 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
VM_BUG_ON(radix_enabled());
slice_range_to_mask(start, len, &mask);
- slice_convert(mm, mask, psize);
+ slice_convert(mm, &mask, psize);
}
#ifdef CONFIG_HUGETLB_PAGE
@@ -772,9 +779,9 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
#if 0 /* too verbose */
slice_dbg("is_hugepage_only_range(mm=%p, addr=%lx, len=%lx)\n",
mm, addr, len);
- slice_print_mask(" mask", mask);
- slice_print_mask(" available", available);
+ slice_print_mask(" mask", &mask);
+ slice_print_mask(" available", &available);
#endif
- return !slice_check_fit(mm, mask, available);
+ return !slice_check_fit(mm, &mask, &available);
}
#endif
--
2.15.1
^ permalink raw reply related
* [RFC PATCH 0/5] powerpc/mm/slice: improve slice speed and stack use
From: Nicholas Piggin @ 2018-02-10 8:11 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
This series intends to improve performance and reduce stack
consumption in the slice allocation code. It does it by keeping slice
masks in the mm_context rather than compute them for each allocation,
and by reducing bitmaps and slice_masks from stacks, using pointers
instead where possible.
checkstack.pl gives, before:
0x00000de4 slice_get_unmapped_area [slice.o]: 656
0x00001b4c is_hugepage_only_range [slice.o]: 512
0x0000075c slice_find_area_topdown [slice.o]: 416
0x000004c8 slice_find_area_bottomup.isra.1 [slice.o]: 272
0x00001aa0 slice_set_range_psize [slice.o]: 240
0x00000a64 slice_find_area [slice.o]: 176
0x00000174 slice_check_fit [slice.o]: 112
after:
0x00000d70 slice_get_unmapped_area [slice.o]: 320
0x000008f8 slice_find_area [slice.o]: 144
0x00001860 slice_set_range_psize [slice.o]: 144
0x000018ec is_hugepage_only_range [slice.o]: 144
0x00000750 slice_find_area_bottomup.isra.4 [slice.o]: 128
The benchmark in https://github.com/linuxppc/linux/issues/49 gives, before:
$ time ./slicemask
real 0m20.712s
user 0m5.830s
sys 0m15.105s
after:
$ time ./slicemask
real 0m13.197s
user 0m5.409s
sys 0m7.779s
Thanks,
Nick
Nicholas Piggin (5):
powerpc/mm/slice: pass pointers to struct slice_mask where possible
powerpc/mm/slice: implement a slice mask cache
powerpc/mm/slice: implement slice_check_range_fits
powerpc/mm/slice: Use const pointers to cached slice masks where
possible
powerpc/mm/slice: use the dynamic high slice size to limit bitmap
operations
arch/powerpc/include/asm/book3s/64/mmu.h | 20 +-
arch/powerpc/mm/slice.c | 302 +++++++++++++++++++------------
2 files changed, 204 insertions(+), 118 deletions(-)
--
2.15.1
^ permalink raw reply
* Re: [PATCH v1] PCI: Make PCI_SCAN_ALL_PCIE_DEVS work for Root as well as Downstream Ports
From: Christian Zigotzky @ 2018-02-10 8:05 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: linux-pci, Michael Ellerman, linuxppc-dev, linux-kernel
In-Reply-To: <20171202191827.GB18780@bhelgaas-glaptop.roam.corp.google.com>
Hi All,
The AmigaOne X1000 doesn=E2=80=99t boot anymore since the PCI updates. I hav=
e seen, that the PCI updates are different to the updates below. The code be=
low works but the latest not. Is there a problem with the latest PCI updates=
currently?
Thanks,
Christian
Sent from my iPhone
On 2. Dec 2017, at 20:18, Bjorn Helgaas <helgaas@kernel.org> wrote:
On Fri, Dec 01, 2017 at 06:27:10PM -0600, Bjorn Helgaas wrote:
From: Bjorn Helgaas <bhelgaas@google.com>
PCIe Downstream Ports normally have only a Device 0 below them. To
optimize enumeration, we don't scan for other devices *unless* the
PCI_SCAN_ALL_PCIE_DEVS flag is set by set by quirks or the
"pci=3Dpcie_scan_all" kernel parameter.
Previously PCI_SCAN_ALL_PCIE_DEVS only affected scanning below Switch
Downstream Ports, not Root Ports.
But the "Nemo" system, also known as the AmigaOne X1000, has a PA Semi Root
Port whose link leads to an AMD/ATI SB600 South Bridge. The Root Port is a
PCIe device, of course, but the SB600 contains only conventional PCI
devices with no visible PCIe port.
Simplify and restructure only_one_child() so that we scan for all possible
devices below Root Ports as well as Switch Downstream Ports when
PCI_SCAN_ALL_PCIE_DEVS is set.
This is enough to make Nemo work with "pci=3Dpcie_scan_all". We would also
like to add a quirk to set PCI_SCAN_ALL_PCIE_DEVS automatically on Nemo so
users wouldn't have to use the "pci=3Dpcie_scan_all" parameter, but we don't=
have that yet.
Link: https://lkml.kernel.org/r/CAErSpo55Q8Q=3D5p6_+uu7ahnw+53ibVDNRXxrzRV9Q=
nUr_9EUfw@mail.gmail.com
Link: https://bugzilla.kernel.org/show_bug.cgi?id=3D198057
Reported-and-Tested-by: Christian Zigotzky <chzigotzky@xenosoft.de>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Applied to pci/enumeration for v4.16.
---
drivers/pci/probe.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 14e0ea1ff38b..303c0cb0550c 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -2215,22 +2215,27 @@ static unsigned next_fn(struct pci_bus *bus, struct p=
ci_dev *dev, unsigned fn)
static int only_one_child(struct pci_bus *bus)
{
- struct pci_dev *parent =3D bus->self;
+ struct pci_dev *bridge =3D bus->self;
- if (!parent || !pci_is_pcie(parent))
+ /*
+ * Systems with unusual topologies set PCI_SCAN_ALL_PCIE_DEVS so
+ * we scan for all possible devices, not just Device 0.
+ */
+ if (pci_has_flag(PCI_SCAN_ALL_PCIE_DEVS))
return 0;
- if (pci_pcie_type(parent) =3D=3D PCI_EXP_TYPE_ROOT_PORT)
- return 1;
/*
- * PCIe downstream ports are bridges that normally lead to only a
- * device 0, but if PCI_SCAN_ALL_PCIE_DEVS is set, scan all
- * possible devices, not just device 0. See PCIe spec r3.0,
- * sec 7.3.1.
+ * A PCIe Downstream Port normally leads to a Link with only Device
+ * 0 on it (PCIe spec r3.1, sec 7.3.1). As an optimization, scan
+ * only for Device 0 in that situation.
+ *
+ * Checking has_secondary_link is a hack to identify Downstream
+ * Ports because sometimes Switches are configured such that the
+ * PCIe Port Type labels are backwards.
*/
- if (parent->has_secondary_link &&
- !pci_has_flag(PCI_SCAN_ALL_PCIE_DEVS))
+ if (bridge && pci_is_pcie(bridge) && bridge->has_secondary_link)
return 1;
+
return 0;
}
^ permalink raw reply related
* Re: [PATCH] powerpc/vas: do not set uses_vas for kernel windows
From: Sukadev Bhattiprolu @ 2018-02-10 4:57 UTC (permalink / raw)
To: Nicholas Piggin; +Cc: linuxppc-dev
In-Reply-To: <20180208091838.27992-1-npiggin@gmail.com>
Nicholas Piggin [npiggin@gmail.com] wrote:
> cp_abort is only required or user windows, because kernel context
> must not be preempted between a copy/paste pair.
Yes, that is a good optimization.
>
> Without this patch, the init task gets used_vas set when it runs
> the nx842_powernv_init initcall, which opens windows for kernel
> usage.
>
> used_vas is then never cleared anywhere, so it gets propagated
> into all other tasks. It's a property of the address space, so it
> should really be cleared when a new mm is created (or in dup_mmap
> if the mmaps are marked as VM_DONTCOPY). For now we seem to have
> no such driver, so leave that for another patch.
If the parent process has the paste address mapped, the child inherits
those mappings - so we can't clear the ->used_vas in a process until
it has unmapped all the send windows right?
If VM_DONCOPY is set, then we can clear it.
>
> Cc: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
^ permalink raw reply
* [PATCH 4/4] powerpc/vas: Add a couple of trace points
From: Sukadev Bhattiprolu @ 2018-02-10 3:49 UTC (permalink / raw)
To: Michael Ellerman
Cc: Benjamin Herrenschmidt, mikey, hbabu, linuxppc-dev, linux-kernel
In-Reply-To: <1518234567-24869-1-git-send-email-sukadev@linux.vnet.ibm.com>
Add a couple of trace points in the VAS driver
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
Changelog [v2]
- Make TRACE_INCLUDE_PATH relative to <trace/define_trace.h>
---
arch/powerpc/platforms/powernv/vas-trace.h | 112 ++++++++++++++++++++++++++++
arch/powerpc/platforms/powernv/vas-window.c | 9 +++
2 files changed, 121 insertions(+)
create mode 100644 arch/powerpc/platforms/powernv/vas-trace.h
diff --git a/arch/powerpc/platforms/powernv/vas-trace.h b/arch/powerpc/platforms/powernv/vas-trace.h
new file mode 100644
index 0000000..939d85d
--- /dev/null
+++ b/arch/powerpc/platforms/powernv/vas-trace.h
@@ -0,0 +1,112 @@
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM vas
+
+#if !defined(_VAS_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+
+#define _VAS_TRACE_H
+#include <linux/tracepoint.h>
+#include <linux/sched.h>
+#include <asm/vas.h>
+
+TRACE_EVENT( vas_rx_win_open,
+
+ TP_PROTO(struct task_struct *tsk,
+ int vasid,
+ int cop,
+ struct vas_rx_win_attr *rxattr),
+
+ TP_ARGS(tsk, vasid, cop, rxattr),
+
+ TP_STRUCT__entry(
+ __field(struct task_struct *, tsk)
+ __field(int, pid)
+ __field(int, cop)
+ __field(int, vasid)
+ __field(struct vas_rx_win_attr *, rxattr)
+ __field(int, lnotify_lpid)
+ __field(int, lnotify_pid)
+ __field(int, lnotify_tid)
+ ),
+
+ TP_fast_assign(
+ __entry->pid = tsk->pid;
+ __entry->vasid = vasid;
+ __entry->cop = cop;
+ __entry->lnotify_lpid = rxattr->lnotify_lpid;
+ __entry->lnotify_pid = rxattr->lnotify_pid;
+ __entry->lnotify_tid = rxattr->lnotify_tid;
+ ),
+
+ TP_printk("pid=%d, vasid=%d, cop=%d, lpid=%d, pid=%d, tid=%d",
+ __entry->pid, __entry->vasid, __entry->cop,
+ __entry->lnotify_lpid, __entry->lnotify_pid,
+ __entry->lnotify_tid)
+);
+
+TRACE_EVENT( vas_tx_win_open,
+
+ TP_PROTO(struct task_struct *tsk,
+ int vasid,
+ int cop,
+ struct vas_tx_win_attr *txattr),
+
+ TP_ARGS(tsk, vasid, cop, txattr),
+
+ TP_STRUCT__entry(
+ __field(struct task_struct *, tsk)
+ __field(int, pid)
+ __field(int, cop)
+ __field(int, vasid)
+ __field(struct vas_tx_win_attr *, txattr)
+ __field(int, lpid)
+ __field(int, pidr)
+ ),
+
+ TP_fast_assign(
+ __entry->pid = tsk->pid;
+ __entry->vasid = vasid;
+ __entry->cop = cop;
+ __entry->lpid = txattr->lpid;
+ __entry->pidr = txattr->pidr;
+ ),
+
+ TP_printk("pid=%d, vasid=%d, cop=%d, lpid=%d, pidr=%d",
+ __entry->pid, __entry->vasid, __entry->cop,
+ __entry->lpid, __entry->pidr)
+);
+
+TRACE_EVENT( vas_paste_crb,
+
+ TP_PROTO(struct task_struct *tsk,
+ struct vas_window *win),
+
+ TP_ARGS(tsk, win),
+
+ TP_STRUCT__entry(
+ __field(struct task_struct *, tsk)
+ __field(struct vas_window *, win)
+ __field(int, pid)
+ __field(int, vasid)
+ __field(int, winid)
+ __field(unsigned long, paste_kaddr)
+ ),
+
+ TP_fast_assign(
+ __entry->pid = tsk->pid;
+ __entry->vasid = win->vinst->vas_id;
+ __entry->winid = win->winid;
+ __entry->paste_kaddr = (unsigned long)win->paste_kaddr
+ ),
+
+ TP_printk("pid=%d, vasid=%d, winid=%d, paste_kaddr=0x%016lx\n",
+ __entry->pid, __entry->vasid, __entry->winid,
+ __entry->paste_kaddr)
+);
+
+#endif /* _VAS_TRACE_H */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH ../../arch/powerpc/platforms/powernv
+#define TRACE_INCLUDE_FILE vas-trace
+#include <trace/define_trace.h>
diff --git a/arch/powerpc/platforms/powernv/vas-window.c b/arch/powerpc/platforms/powernv/vas-window.c
index 2b3eb01..6b2de9e 100644
--- a/arch/powerpc/platforms/powernv/vas-window.c
+++ b/arch/powerpc/platforms/powernv/vas-window.c
@@ -21,6 +21,9 @@
#include "vas.h"
#include "copy-paste.h"
+#define CREATE_TRACE_POINTS
+#include "vas-trace.h"
+
/*
* Compute the paste address region for the window @window using the
* ->paste_base_addr and ->paste_win_id_shift we got from device tree.
@@ -880,6 +883,8 @@ struct vas_window *vas_rx_win_open(int vasid, enum vas_cop_type cop,
struct vas_winctx winctx;
struct vas_instance *vinst;
+ trace_vas_rx_win_open(current, vasid, cop, rxattr);
+
if (!rx_win_args_valid(cop, rxattr))
return ERR_PTR(-EINVAL);
@@ -1008,6 +1013,8 @@ struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop,
struct vas_winctx winctx;
struct vas_instance *vinst;
+ trace_vas_tx_win_open(current, vasid, cop, attr);
+
if (!tx_win_args_valid(cop, attr))
return ERR_PTR(-EINVAL);
@@ -1100,6 +1107,8 @@ int vas_paste_crb(struct vas_window *txwin, int offset, bool re)
void *addr;
uint64_t val;
+ trace_vas_paste_crb(current, txwin);
+
/*
* Only NX windows are supported for now and hardware assumes
* report-enable flag is set for NX windows. Ensure software
--
2.7.4
^ permalink raw reply related
* [PATCH RESEND 3/4] powerpc/vas: Remove a stray line in Makefile
From: Sukadev Bhattiprolu @ 2018-02-10 3:49 UTC (permalink / raw)
To: Michael Ellerman
Cc: Benjamin Herrenschmidt, mikey, hbabu, linuxppc-dev, linux-kernel
In-Reply-To: <1518234567-24869-1-git-send-email-sukadev@linux.vnet.ibm.com>
Remove a bogus line from arch/powerpc/platforms/powernv/Makefile that
was added by commit ece4e51 ("powerpc/vas: Export HVWC to debugfs").
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
---
arch/powerpc/platforms/powernv/Makefile | 1 -
1 file changed, 1 deletion(-)
diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
index 6c9d519..703a350 100644
--- a/arch/powerpc/platforms/powernv/Makefile
+++ b/arch/powerpc/platforms/powernv/Makefile
@@ -16,5 +16,4 @@ obj-$(CONFIG_OPAL_PRD) += opal-prd.o
obj-$(CONFIG_PERF_EVENTS) += opal-imc.o
obj-$(CONFIG_PPC_MEMTRACE) += memtrace.o
obj-$(CONFIG_PPC_VAS) += vas.o vas-window.o vas-debug.o
-obj-$(CONFIG_PPC_FTW) += nx-ftw.o
obj-$(CONFIG_OCXL_BASE) += ocxl.o
--
2.7.4
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox