* Re: [PATCH 2/2] powerpc/perf: Fix the kernel address leak to userspace via SDAR
From: Madhavan Srinivasan @ 2018-03-07 4:53 UTC (permalink / raw)
To: Naveen N. Rao, mpe; +Cc: linuxppc-dev
In-Reply-To: <1520237954.soawktcmbp.naveen@linux.ibm.com>
On Monday 05 March 2018 01:51 PM, Naveen N. Rao wrote:
> Madhavan Srinivasan wrote:
>> Sampled Data Address Register (SDAR) is a 64-bit
>> register that contains the effective address of
>> the storage operand of an instruction that was
>> being executed, possibly out-of-order, at or around
>> the time that the Performance Monitor alert occurred.
>>
>> In certain scenario SDAR happen to contain the kernel
>> address even for userspace only sampling. Add checks
>> to prevent it.
>>
>> Signed-off-by: Madhavan Srinivasan <maddy@linux.vnet.ibm.com>
>> ---
>> arch/powerpc/perf/core-book3s.c | 11 ++++++++---
>> 1 file changed, 8 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/powerpc/perf/core-book3s.c
>> b/arch/powerpc/perf/core-book3s.c
>> index 337db5831749..c4525323d691 100644
>> --- a/arch/powerpc/perf/core-book3s.c
>> +++ b/arch/powerpc/perf/core-book3s.c
>> @@ -95,7 +95,7 @@ static inline unsigned long perf_ip_adjust(struct
>> pt_regs *regs)
>> {
>> return 0;
>> }
>> -static inline void perf_get_data_addr(struct pt_regs *regs, u64
>> *addrp) { }
>> +static inline void perf_get_data_addr(struct pt_regs *regs, u64
>> *addrp, struct perf_event *event) { }
>> static inline u32 perf_get_misc_flags(struct pt_regs *regs)
>> {
>> return 0;
>> @@ -174,7 +174,7 @@ static inline unsigned long perf_ip_adjust(struct
>> pt_regs *regs)
>> * pointed to by SIAR; this is indicated by the
>> [POWER6_]MMCRA_SDSYNC, the
>> * [POWER7P_]MMCRA_SDAR_VALID bit in MMCRA, or the SDAR_VALID bit in
>> SIER.
>> */
>> -static inline void perf_get_data_addr(struct pt_regs *regs, u64 *addrp)
>> +static inline void perf_get_data_addr(struct pt_regs *regs, u64
>> *addrp, struct perf_event *event)
>> {
>> unsigned long mmcra = regs->dsisr;
>> bool sdar_valid;
>> @@ -198,6 +198,11 @@ static inline void perf_get_data_addr(struct
>> pt_regs *regs, u64 *addrp)
>>
>> if (!(mmcra & MMCRA_SAMPLE_ENABLE) || sdar_valid)
>> *addrp = mfspr(SPRN_SDAR);
>> +
>> + if (perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN) &&
>> + (event->attr.exclude_kernel || event->attr.exclude_hv) &&
>
> I may be missing something, but if !capable(CAP_SYS_ADMIN), should we
> still check the exclude_kernel/exclude_hv fields in the event
> attribute? Aren't those user controlled?
>
Yes that right. But i also want to handle the case when we sampling only
for userspace even with higher privilege level. May be I should handle
that as a separate patch.
I will respin this patch to check only for the privilege level and
change the commit message accordingly.
Thanks for review
Maddy
> - Naveen
>
>> + is_kernel_addr(mfspr(SPRN_SDAR)))
>> + *addrp = 0;
>> }
>>
>> static bool regs_sihv(struct pt_regs *regs)
>> @@ -2054,7 +2059,7 @@ static void record_and_restart(struct
>> perf_event *event, unsigned long val,
>>
>> if (event->attr.sample_type &
>> (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR))
>> - perf_get_data_addr(regs, &data.addr);
>> + perf_get_data_addr(regs, &data.addr, event);
>>
>> if (event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK) {
>> struct cpu_hw_events *cpuhw;
>> --
>> 2.7.4
>>
>>
^ permalink raw reply
* Re: [PATCH v2 04/38] cxlflash: Introduce OCXL backend
From: Andrew Donnellan @ 2018-03-07 4:44 UTC (permalink / raw)
To: Uma Krishnan, linux-scsi, James Bottomley, Martin K. Petersen,
Matthew R. Ochs, Manoj N. Kumar
Cc: linuxppc-dev, Frederic Barrat, Christophe Lombard
In-Reply-To: <1519683614-16894-1-git-send-email-ukrishn@linux.vnet.ibm.com>
On 27/02/18 09:20, Uma Krishnan wrote:
> Add initial infrastructure to support a new cxlflash transport, OCXL.
>
> Claim a dependency on OCXL and add a new file, ocxl_hw.c, which will host
> the backend routines that are specific to OCXL.
>
> Signed-off-by: Uma Krishnan <ukrishn@linux.vnet.ibm.com>
> Acked-by: Matthew R. Ochs <mrochs@linux.vnet.ibm.com>
Per my response to the first version of this series, at some point we
might want to make this not depend on OCXL in all cases, but I don't
feel strongly on that.
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
--
Andrew Donnellan OzLabs, ADL Canberra
andrew.donnellan@au1.ibm.com IBM Australia Limited
^ permalink raw reply
* [PATCH V3 3/3] powerpc/mm/hash: Don't memset pgd table if not needed
From: Aneesh Kumar K.V @ 2018-03-07 4:42 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <20180307044204.8904-1-aneesh.kumar@linux.vnet.ibm.com>
We need to zero-out pgd table only if we share the slab cache with pud/pmd
level caches. With the support of 4PB, we don't share the slab cache anymore.
Instead of removing the code completely hide it within an #ifdef. We don't need
to do this with any other page table level, because they all allocate table
of double the size and we take of initializing the first half corrrectly during
page table zap.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/64/pgalloc.h | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
index 4746bc68d446..07f0dbac479f 100644
--- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
@@ -80,8 +80,19 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
pgd = kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE),
pgtable_gfp_flags(mm, GFP_KERNEL));
+ /*
+ * With hugetlb, we don't clear the second half of the page table.
+ * If we share the same slab cache with the pmd or pud level table,
+ * we need to make sure we zero out the full table on alloc.
+ * With 4K we don't store slot in the second half. Hence we don't
+ * need to do this for 4k.
+ */
+#if (H_PGD_INDEX_SIZE == H_PUD_CACHE_INDEX) || \
+ (H_PGD_INDEX_SIZE == H_PMD_CACHE_INDEX)
+#if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_PPC_64K_PAGES)
memset(pgd, 0, PGD_TABLE_SIZE);
-
+#endif
+#endif
return pgd;
}
--
2.14.3
^ permalink raw reply related
* [PATCH V3 2/3] powerpc/mm/hash64: Increase the VA range
From: Aneesh Kumar K.V @ 2018-03-07 4:42 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <20180307044204.8904-1-aneesh.kumar@linux.vnet.ibm.com>
This patch increase the max virtual address value to 4PB. With 4K page size
config we continue to limit ourself to 64TB.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/64/hash-64k.h | 2 +-
arch/powerpc/include/asm/processor.h | 9 ++++++++-
arch/powerpc/mm/init_64.c | 6 ------
arch/powerpc/mm/pgtable_64.c | 5 -----
4 files changed, 9 insertions(+), 13 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/hash-64k.h b/arch/powerpc/include/asm/book3s/64/hash-64k.h
index 0ee0fc1ad675..02098d7fe177 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-64k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-64k.h
@@ -4,7 +4,7 @@
#define H_PTE_INDEX_SIZE 8
#define H_PMD_INDEX_SIZE 10
-#define H_PUD_INDEX_SIZE 7
+#define H_PUD_INDEX_SIZE 10
#define H_PGD_INDEX_SIZE 8
/*
* No of address bits below which we use the default context
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 70d65b482504..a621a068880a 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -109,6 +109,13 @@ void release_thread(struct task_struct *);
#define TASK_SIZE_64TB (0x0000400000000000UL)
#define TASK_SIZE_128TB (0x0000800000000000UL)
#define TASK_SIZE_512TB (0x0002000000000000UL)
+#define TASK_SIZE_1PB (0x0004000000000000UL)
+#define TASK_SIZE_2PB (0x0008000000000000UL)
+/*
+ * With 52 bits in the address we can support
+ * upto 4PB of range.
+ */
+#define TASK_SIZE_4PB (0x0010000000000000UL)
/*
* For now 512TB is only supported with book3s and 64K linux page size.
@@ -117,7 +124,7 @@ void release_thread(struct task_struct *);
/*
* Max value currently used:
*/
-#define TASK_SIZE_USER64 TASK_SIZE_512TB
+#define TASK_SIZE_USER64 TASK_SIZE_4PB
#define DEFAULT_MAP_WINDOW_USER64 TASK_SIZE_128TB
#define TASK_CONTEXT_SIZE TASK_SIZE_512TB
#else
diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index fdb424a29f03..63470b06c502 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -68,12 +68,6 @@
#include "mmu_decl.h"
-#ifdef CONFIG_PPC_BOOK3S_64
-#if H_PGTABLE_RANGE > USER_VSID_RANGE
-#warning Limited user VSID range means pagetable space is wasted
-#endif
-#endif /* CONFIG_PPC_BOOK3S_64 */
-
phys_addr_t memstart_addr = ~0;
EXPORT_SYMBOL_GPL(memstart_addr);
phys_addr_t kernstart_addr;
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index 28c980eb4422..16636bdf3331 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -57,11 +57,6 @@
#include "mmu_decl.h"
-#ifdef CONFIG_PPC_BOOK3S_64
-#if TASK_SIZE_USER64 > (1UL << (ESID_BITS + SID_SHIFT))
-#error TASK_SIZE_USER64 exceeds user VSID range
-#endif
-#endif
#ifdef CONFIG_PPC_BOOK3S_64
/*
--
2.14.3
^ permalink raw reply related
* [PATCH V3 1/3] powerpc/mm: Add support for handling > 512TB address in SLB miss
From: Aneesh Kumar K.V @ 2018-03-07 4:42 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <20180307044204.8904-1-aneesh.kumar@linux.vnet.ibm.com>
For address above 512TB we allocate additional mmu context. To make it all
easy address above 512TB is handled with IR/DR=1 and with stack frame setup.
We do the additional context allocation in SLB miss handler. If the context is
not allocated, we enable interrupts and allocate the context and retry the
access which will again result in a SLB miss.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/64/hash-4k.h | 6 ++
arch/powerpc/include/asm/book3s/64/hash-64k.h | 5 +
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 6 +-
arch/powerpc/include/asm/book3s/64/mmu.h | 24 +++++
arch/powerpc/include/asm/processor.h | 7 ++
arch/powerpc/kernel/exceptions-64s.S | 12 ++-
arch/powerpc/mm/copro_fault.c | 2 +-
arch/powerpc/mm/hash_utils_64.c | 4 +-
arch/powerpc/mm/mmu_context_book3s64.c | 17 ++-
arch/powerpc/mm/pgtable-hash64.c | 2 +-
arch/powerpc/mm/slb.c | 146 ++++++++++++++++++++++++++
arch/powerpc/mm/slb_low.S | 6 +-
arch/powerpc/mm/tlb_hash64.c | 2 +-
13 files changed, 224 insertions(+), 15 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/hash-4k.h b/arch/powerpc/include/asm/book3s/64/hash-4k.h
index 67c5475311ee..af2ba9875f18 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-4k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-4k.h
@@ -11,6 +11,12 @@
#define H_PUD_INDEX_SIZE 9
#define H_PGD_INDEX_SIZE 9
+/*
+ * No of address bits below which we use the default context
+ * for slb allocation. For 4k this is 64TB.
+ */
+#define H_BITS_FIRST_CONTEXT 46
+
#ifndef __ASSEMBLY__
#define H_PTE_TABLE_SIZE (sizeof(pte_t) << H_PTE_INDEX_SIZE)
#define H_PMD_TABLE_SIZE (sizeof(pmd_t) << H_PMD_INDEX_SIZE)
diff --git a/arch/powerpc/include/asm/book3s/64/hash-64k.h b/arch/powerpc/include/asm/book3s/64/hash-64k.h
index 3bcf269f8f55..0ee0fc1ad675 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-64k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-64k.h
@@ -6,6 +6,11 @@
#define H_PMD_INDEX_SIZE 10
#define H_PUD_INDEX_SIZE 7
#define H_PGD_INDEX_SIZE 8
+/*
+ * No of address bits below which we use the default context
+ * for slb allocation. For 64k this is 512TB.
+ */
+#define H_BITS_FIRST_CONTEXT 49
/*
* 64k aligned address free up few of the lower bits of RPN for us
diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
index 50ed64fba4ae..8ee83f6e9c84 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
@@ -691,8 +691,8 @@ static inline int user_segment_size(unsigned long addr)
return MMU_SEGSIZE_256M;
}
-static inline unsigned long get_vsid(unsigned long context, unsigned long ea,
- int ssize)
+static inline unsigned long __get_vsid(unsigned long context, unsigned long ea,
+ int ssize)
{
unsigned long va_bits = VA_BITS;
unsigned long vsid_bits;
@@ -744,7 +744,7 @@ static inline unsigned long get_kernel_vsid(unsigned long ea, int ssize)
*/
context = (ea >> 60) - KERNEL_REGION_CONTEXT_OFFSET;
- return get_vsid(context, ea, ssize);
+ return __get_vsid(context, ea, ssize);
}
unsigned htab_shift_for_mem_size(unsigned long mem_size);
diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
index 777778579305..fbf9081e6ece 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu.h
@@ -92,6 +92,12 @@ struct slice_mask {
typedef struct {
mm_context_id_t id;
+ /*
+ * One context for each 512TB above the first 512TB.
+ * First 512TB context is saved in id and is also used
+ * as PIDR.
+ */
+ mm_context_id_t extended_id[(TASK_SIZE_USER64/TASK_CONTEXT_SIZE) - 1];
u16 user_psize; /* page size index */
/* Number of bits in the mm_cpumask */
@@ -193,5 +199,23 @@ extern void radix_init_pseries(void);
static inline void radix_init_pseries(void) { };
#endif
+static inline int get_esid_context(mm_context_t *ctx, unsigned long ea)
+{
+ int index = ea >> H_BITS_FIRST_CONTEXT;
+
+ if (index == 0)
+ return ctx->id;
+ return ctx->extended_id[--index];
+}
+
+static inline unsigned long get_user_vsid(mm_context_t *ctx,
+ unsigned long ea, int ssize)
+{
+ unsigned long context = get_esid_context(ctx, ea);
+
+ return __get_vsid(context, ea, ssize);
+}
+
+
#endif /* __ASSEMBLY__ */
#endif /* _ASM_POWERPC_BOOK3S_64_MMU_H_ */
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 01299cdc9806..70d65b482504 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -119,9 +119,16 @@ void release_thread(struct task_struct *);
*/
#define TASK_SIZE_USER64 TASK_SIZE_512TB
#define DEFAULT_MAP_WINDOW_USER64 TASK_SIZE_128TB
+#define TASK_CONTEXT_SIZE TASK_SIZE_512TB
#else
#define TASK_SIZE_USER64 TASK_SIZE_64TB
#define DEFAULT_MAP_WINDOW_USER64 TASK_SIZE_64TB
+/*
+ * We don't need allocate extended context id for 4K
+ * page size. We limit max address on this config to
+ * 64TB.
+ */
+#define TASK_CONTEXT_SIZE TASK_SIZE_64TB
#endif
/*
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 3ac87e53b3da..166b8c0f1830 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -620,8 +620,12 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX)
ld r10,PACA_EXSLB+EX_LR(r13)
lwz r9,PACA_EXSLB+EX_CCR(r13) /* get saved CR */
mtlr r10
+ /*
+ * Large address, check whether we have to allocate new
+ * contexts.
+ */
+ beq- 8f
- beq- 8f /* if bad address, make full stack frame */
bne- cr5,2f /* if unrecoverable exception, oops */
@@ -685,7 +689,7 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX)
mr r3,r12
mfspr r11,SPRN_SRR0
mfspr r12,SPRN_SRR1
- LOAD_HANDLER(r10,bad_addr_slb)
+ LOAD_HANDLER(r10, multi_context_slb)
mtspr SPRN_SRR0,r10
ld r10,PACAKMSR(r13)
mtspr SPRN_SRR1,r10
@@ -700,7 +704,7 @@ EXC_COMMON_BEGIN(unrecov_slb)
bl unrecoverable_exception
b 1b
-EXC_COMMON_BEGIN(bad_addr_slb)
+EXC_COMMON_BEGIN(multi_context_slb)
EXCEPTION_PROLOG_COMMON(0x380, PACA_EXSLB)
RECONCILE_IRQ_STATE(r10, r11)
ld r3, PACA_EXSLB+EX_DAR(r13)
@@ -710,7 +714,7 @@ EXC_COMMON_BEGIN(bad_addr_slb)
std r10, _TRAP(r1)
2: bl save_nvgprs
addi r3, r1, STACK_FRAME_OVERHEAD
- bl slb_miss_bad_addr
+ bl handle_multi_context_slb_miss
b ret_from_except
EXC_REAL_BEGIN(hardware_interrupt, 0x500, 0x100)
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index 697b70ad1195..7d0945bd3a61 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -112,7 +112,7 @@ int copro_calculate_slb(struct mm_struct *mm, u64 ea, struct copro_slb *slb)
return 1;
psize = get_slice_psize(mm, ea);
ssize = user_segment_size(ea);
- vsid = get_vsid(mm->context.id, ea, ssize);
+ vsid = get_user_vsid(&mm->context, ea, ssize);
vsidkey = SLB_VSID_USER;
break;
case VMALLOC_REGION_ID:
diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
index b578148d89e6..f62325d4f5f5 100644
--- a/arch/powerpc/mm/hash_utils_64.c
+++ b/arch/powerpc/mm/hash_utils_64.c
@@ -1261,7 +1261,7 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea,
}
psize = get_slice_psize(mm, ea);
ssize = user_segment_size(ea);
- vsid = get_vsid(mm->context.id, ea, ssize);
+ vsid = get_user_vsid(&mm->context, ea, ssize);
break;
case VMALLOC_REGION_ID:
vsid = get_kernel_vsid(ea, mmu_kernel_ssize);
@@ -1526,7 +1526,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea,
/* Get VSID */
ssize = user_segment_size(ea);
- vsid = get_vsid(mm->context.id, ea, ssize);
+ vsid = get_user_vsid(&mm->context, ea, ssize);
if (!vsid)
return;
/*
diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c
index 80acad52b006..1e3960fa2df2 100644
--- a/arch/powerpc/mm/mmu_context_book3s64.c
+++ b/arch/powerpc/mm/mmu_context_book3s64.c
@@ -178,6 +178,19 @@ void __destroy_context(int context_id)
}
EXPORT_SYMBOL_GPL(__destroy_context);
+void destroy_extended_context(mm_context_t *ctx)
+{
+ int index, context_id;
+
+ spin_lock(&mmu_context_lock);
+ for (index = 1; index < (TASK_SIZE_USER64/TASK_CONTEXT_SIZE); index++) {
+ context_id = ctx->extended_id[index - 1];
+ if (context_id)
+ ida_remove(&mmu_context_ida, context_id);
+ }
+ spin_unlock(&mmu_context_lock);
+}
+
#ifdef CONFIG_PPC_64K_PAGES
static void destroy_pagetable_page(struct mm_struct *mm)
{
@@ -213,8 +226,10 @@ void destroy_context(struct mm_struct *mm)
#endif
if (radix_enabled())
WARN_ON(process_tb[mm->context.id].prtb0 != 0);
- else
+ else {
subpage_prot_free(mm);
+ destroy_extended_context(&mm->context);
+ }
destroy_pagetable_page(mm);
__destroy_context(mm->context.id);
mm->context.id = MMU_NO_CONTEXT;
diff --git a/arch/powerpc/mm/pgtable-hash64.c b/arch/powerpc/mm/pgtable-hash64.c
index 469808e77e58..a87b18cf6749 100644
--- a/arch/powerpc/mm/pgtable-hash64.c
+++ b/arch/powerpc/mm/pgtable-hash64.c
@@ -320,7 +320,7 @@ void hpte_do_hugepage_flush(struct mm_struct *mm, unsigned long addr,
if (!is_kernel_addr(addr)) {
ssize = user_segment_size(addr);
- vsid = get_vsid(mm->context.id, addr, ssize);
+ vsid = get_user_vsid(&mm->context, addr, ssize);
WARN_ON(vsid == 0);
} else {
vsid = get_kernel_vsid(addr, mmu_kernel_ssize);
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 13cfe413b40d..e5bde73c62df 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -23,6 +23,7 @@
#include <asm/smp.h>
#include <linux/compiler.h>
#include <linux/mm_types.h>
+#include <linux/context_tracking.h>
#include <asm/udbg.h>
#include <asm/code-patching.h>
@@ -340,3 +341,148 @@ void slb_initialize(void)
asm volatile("isync":::"memory");
}
+
+/*
+ * Only handle insert of 1TB slb entries.
+ */
+static void insert_slb_entry(unsigned long vsid, unsigned long ea,
+ int bpsize, int ssize)
+{
+ int slb_cache_index;
+ unsigned long flags;
+ enum slb_index index;
+ unsigned long vsid_data, esid_data;
+
+ /*
+ * We are irq disabled, hence should be safe
+ * to access PACA.
+ */
+ index = get_paca()->stab_rr;
+ /*
+ * simple round roubin replacement of slb.
+ */
+ if (index < mmu_slb_size)
+ index++;
+ else
+ index = SLB_NUM_BOLTED;
+ get_paca()->stab_rr = index;
+
+ flags = SLB_VSID_USER | mmu_psize_defs[bpsize].sllp;
+ vsid_data = (vsid << SLB_VSID_SHIFT_1T) | flags |
+ ((unsigned long) ssize << SLB_VSID_SSIZE_SHIFT);
+ esid_data = mk_esid_data(ea, mmu_highuser_ssize, index);
+
+ asm volatile("slbmte %0, %1" : : "r" (vsid_data), "r" (esid_data)
+ : "memory");
+ /*
+ * Now update slb cache entries
+ */
+ slb_cache_index = get_paca()->slb_cache_ptr;
+ if (slb_cache_index < SLB_CACHE_ENTRIES) {
+ /*
+ * We have space in slb cache for optimized switch_slb().
+ * Top 36 bits from esid_data as per ISA
+ */
+ get_paca()->slb_cache[slb_cache_index++] = esid_data >> 28;
+ }
+ /*
+ * if we are full, just increment and return.
+ */
+ get_paca()->slb_cache_ptr++;
+}
+
+static void alloc_extended_context(struct mm_struct *mm, unsigned long ea)
+{
+ int context_id;
+
+ int index = (ea >> H_BITS_FIRST_CONTEXT) - 1;
+
+ /*
+ * we need to do locking only here. If this value was not set before
+ * we will have taken an SLB miss and will reach here. The value will
+ * be either 0 or a valid extended context. We need to make sure two
+ * parallel SLB miss don't end up allocating extended_context for the
+ * same range. The locking below ensures that. For now we take the
+ * heavy mmap_sem. But can be changed to per mm_context_t custom lock
+ * if needed.
+ */
+ down_read(&mm->mmap_sem);
+ context_id = hash__alloc_context_id();
+ if (context_id < 0) {
+ up_read(&mm->mmap_sem);
+ pagefault_out_of_memory();
+ return;
+ }
+ /* Check for parallel allocation after holding lock */
+ if (!mm->context.extended_id[index])
+ mm->context.extended_id[index] = context_id;
+ else
+ __destroy_context(context_id);
+ up_read(&mm->mmap_sem);
+}
+
+static void __handle_multi_context_slb_miss(struct pt_regs *regs,
+ unsigned long ea)
+{
+ int context, bpsize;
+ unsigned long vsid;
+ struct mm_struct *mm = current->mm;
+
+ context = get_esid_context(&mm->context, ea);
+ if (!context) {
+ /*
+ * haven't allocated context yet for this range.
+ * Enable irq and allo context and return. We will
+ * take an slb miss on this again and come here with
+ * allocated context.
+ */
+ /* We restore the interrupt state now */
+ if (!arch_irq_disabled_regs(regs))
+ local_irq_enable();
+ return alloc_extended_context(mm, ea);
+ }
+ /*
+ * We are always above 1TB, hence use high user segment size.
+ */
+ vsid = __get_vsid(context, ea, mmu_highuser_ssize);
+ bpsize = get_slice_psize(mm, ea);
+
+ insert_slb_entry(vsid, ea, bpsize, mmu_highuser_ssize);
+}
+
+/*
+ * exception_enter() handling? FIXME!!
+ */
+void handle_multi_context_slb_miss(struct pt_regs *regs)
+{
+ enum ctx_state prev_state = exception_enter();
+ unsigned long ea = regs->dar;
+
+ /*
+ * Kernel always runs with single context. Hence
+ * anything that request for multi context is
+ * considered bad slb request.
+ */
+ if (!user_mode(regs))
+ return bad_page_fault(regs, ea, SIGSEGV);
+
+ if (REGION_ID(ea) != USER_REGION_ID)
+ goto slb_bad_addr;
+ /*
+ * Are we beyound what the page table layout support ?
+ */
+ if ((ea & ~REGION_MASK) >= H_PGTABLE_RANGE)
+ goto slb_bad_addr;
+
+ /* Lower address should be handled by asm code */
+ if (ea < (1UL << H_BITS_FIRST_CONTEXT))
+ goto slb_bad_addr;
+
+ __handle_multi_context_slb_miss(regs, ea);
+ exception_exit(prev_state);
+ return;
+
+slb_bad_addr:
+ _exception(SIGSEGV, regs, SEGV_BNDERR, ea);
+ exception_exit(prev_state);
+}
diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S
index 2c7c717fd2ea..c66cb06e73a1 100644
--- a/arch/powerpc/mm/slb_low.S
+++ b/arch/powerpc/mm/slb_low.S
@@ -75,10 +75,12 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_68_BIT_VA)
*/
_GLOBAL(slb_allocate)
/*
- * check for bad kernel/user address
+ * Check for address range for which we need to handle multi context. For
+ * the default context we allocate the slb via the fast path. For large
+ * address we branch out to C-code and look at additional context allocated.
* (ea & ~REGION_MASK) >= PGTABLE_RANGE
*/
- rldicr. r9,r3,4,(63 - H_PGTABLE_EADDR_SIZE - 4)
+ rldicr. r9,r3,4,(63 - H_BITS_FIRST_CONTEXT - 4)
bne- 8f
srdi r9,r3,60 /* get region */
diff --git a/arch/powerpc/mm/tlb_hash64.c b/arch/powerpc/mm/tlb_hash64.c
index 9b23f12e863c..87d71dd25441 100644
--- a/arch/powerpc/mm/tlb_hash64.c
+++ b/arch/powerpc/mm/tlb_hash64.c
@@ -89,7 +89,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr,
/* Build full vaddr */
if (!is_kernel_addr(addr)) {
ssize = user_segment_size(addr);
- vsid = get_vsid(mm->context.id, addr, ssize);
+ vsid = get_user_vsid(&mm->context, addr, ssize);
} else {
vsid = get_kernel_vsid(addr, mmu_kernel_ssize);
ssize = mmu_kernel_ssize;
--
2.14.3
^ permalink raw reply related
* [PATCH V3 0/3] Add support for 4PB virtual address space on hash
From: Aneesh Kumar K.V @ 2018-03-07 4:42 UTC (permalink / raw)
To: benh, paulus, mpe; +Cc: linuxppc-dev, Aneesh Kumar K.V
This patch series extended the max virtual address space value from 512TB
to 4PB with 64K page size. We do that by allocating one vsid context for
each 512TB range. More details of that is explained in patch 3.
Changes from V2:
* Rebased on top of slice_mask series from Nick Piggin
* Fixed segfault when mmap with 512TB hint address
Aneesh Kumar K.V (3):
powerpc/mm: Add support for handling > 512TB address in SLB miss
powerpc/mm/hash64: Increase the VA range
powerpc/mm/hash: Don't memset pgd table if not needed
arch/powerpc/include/asm/book3s/64/hash-4k.h | 6 ++
arch/powerpc/include/asm/book3s/64/hash-64k.h | 7 +-
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 6 +-
arch/powerpc/include/asm/book3s/64/mmu.h | 24 +++++
arch/powerpc/include/asm/book3s/64/pgalloc.h | 13 ++-
arch/powerpc/include/asm/processor.h | 16 ++-
arch/powerpc/kernel/exceptions-64s.S | 12 ++-
arch/powerpc/mm/copro_fault.c | 2 +-
arch/powerpc/mm/hash_utils_64.c | 4 +-
arch/powerpc/mm/init_64.c | 6 --
arch/powerpc/mm/mmu_context_book3s64.c | 17 ++-
arch/powerpc/mm/pgtable-hash64.c | 2 +-
arch/powerpc/mm/pgtable_64.c | 5 -
arch/powerpc/mm/slb.c | 146 ++++++++++++++++++++++++++
arch/powerpc/mm/slb_low.S | 6 +-
arch/powerpc/mm/tlb_hash64.c | 2 +-
16 files changed, 245 insertions(+), 29 deletions(-)
--
2.14.3
^ permalink raw reply
* Re: [PATCH kernel] powerpc/mm: Fix typo in comments
From: Alexey Kardashevskiy @ 2018-03-07 3:49 UTC (permalink / raw)
To: linuxppc-dev
In-Reply-To: <20180201050725.30247-1-aik@ozlabs.ru>
On 01/02/18 16:07, Alexey Kardashevskiy wrote:
> Fixes: 912cc87a6 "powerpc/mm/radix: Add LPID based tlb flush helpers"
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Ping?
> ---
> arch/powerpc/mm/tlb-radix.c | 14 +++++++-------
> 1 file changed, 7 insertions(+), 7 deletions(-)
>
> diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c
> index 71d1b19..001c1f6 100644
> --- a/arch/powerpc/mm/tlb-radix.c
> +++ b/arch/powerpc/mm/tlb-radix.c
> @@ -98,7 +98,7 @@ static inline void __tlbiel_pid(unsigned long pid, int set,
> rb |= set << PPC_BITLSHIFT(51);
> rs = ((unsigned long)pid) << PPC_BITLSHIFT(31);
> prs = 1; /* process scoped */
> - r = 1; /* raidx format */
> + r = 1; /* radix format */
>
> asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1)
> : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory");
> @@ -112,7 +112,7 @@ static inline void __tlbie_pid(unsigned long pid, unsigned long ric)
> rb = PPC_BIT(53); /* IS = 1 */
> rs = pid << PPC_BITLSHIFT(31);
> prs = 1; /* process scoped */
> - r = 1; /* raidx format */
> + r = 1; /* radix format */
>
> asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1)
> : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory");
> @@ -164,7 +164,7 @@ static inline void __tlbiel_va(unsigned long va, unsigned long pid,
> rb |= ap << PPC_BITLSHIFT(58);
> rs = pid << PPC_BITLSHIFT(31);
> prs = 1; /* process scoped */
> - r = 1; /* raidx format */
> + r = 1; /* radix format */
>
> asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1)
> : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory");
> @@ -212,7 +212,7 @@ static inline void __tlbie_va(unsigned long va, unsigned long pid,
> rb |= ap << PPC_BITLSHIFT(58);
> rs = pid << PPC_BITLSHIFT(31);
> prs = 1; /* process scoped */
> - r = 1; /* raidx format */
> + r = 1; /* radix format */
>
> asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1)
> : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory");
> @@ -615,7 +615,7 @@ void radix__flush_tlb_lpid_va(unsigned long lpid, unsigned long gpa,
> rb |= ap << PPC_BITLSHIFT(58);
> rs = lpid & ((1UL << 32) - 1);
> prs = 0; /* process scoped */
> - r = 1; /* raidx format */
> + r = 1; /* radix format */
>
> asm volatile("ptesync": : :"memory");
> asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1)
> @@ -633,7 +633,7 @@ void radix__flush_tlb_lpid(unsigned long lpid)
> rb = 0x2 << PPC_BITLSHIFT(53); /* IS = 2 */
> rs = lpid & ((1UL << 32) - 1);
> prs = 0; /* partition scoped */
> - r = 1; /* raidx format */
> + r = 1; /* radix format */
>
> asm volatile("ptesync": : :"memory");
> asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1)
> @@ -657,7 +657,7 @@ void radix__flush_tlb_all(void)
>
> rb = 0x3 << PPC_BITLSHIFT(53); /* IS = 3 */
> prs = 0; /* partition scoped */
> - r = 1; /* raidx format */
> + r = 1; /* radix format */
> rs = 1 & ((1UL << 32) - 1); /* any LPID value to flush guest mappings */
>
> asm volatile("ptesync": : :"memory");
>
--
Alexey
^ permalink raw reply
* Re: [PATCH kernel] powerpc/npu: Do not try invalidating 32bit table when 64bit table is enabled
From: Alexey Kardashevskiy @ 2018-03-07 3:40 UTC (permalink / raw)
To: linuxppc-dev; +Cc: David Gibson, Alistair Popple, Russell Currey
In-Reply-To: <20180213055135.25639-1-aik@ozlabs.ru>
On 13/02/18 16:51, Alexey Kardashevskiy wrote:
> GPUs and the corresponding NVLink bridges get different PEs as they have
> separate translation validation entries (TVEs). We put these PEs to
> the same IOMMU group so they cannot be passed through separately.
> So the iommu_table_group_ops::set_window/unset_window for GPUs do set
> tables to the NPU PEs as well which means that iommu_table's list of
> attached PEs (iommu_table_group_link) has both GPU and NPU PEs linked.
> This list is used for TCE cache invalidation.
>
> The problem is that NPU PE has just a single TVE and can be programmed
> to point to 32bit or 64bit windows while GPU PE has two (as any other PCI
> device). So we end up having an 32bit iommu_table struct linked to both
> PEs even though only the 64bit TCE table cache can be invalidated on NPU.
> And a relatively recent skiboot detects this and prints errors.
>
> This changes GPU's iommu_table_group_ops::set_window/unset_window to make
> sure that NPU PE is only linked to the table actually used by the hardware.
> If there are two tables used by an IOMMU group, the NPU PE will use
> the last programmed one which with the current use scenarios is expected
> to be a 64bit one.
>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> --
>
> Do we need BUG_ON(IOMMU_TABLE_GROUP_MAX_TABLES != 2)?
Ping?
>
>
> This is an example for:
>
> 0004:04:00.0 3D: NVIDIA Corporation Device 1db1 (rev a1)
> 0006:00:00.0 Bridge: IBM Device 04ea (rev 01)
> 0006:00:00.1 Bridge: IBM Device 04ea (rev 01)
>
> Before the patch (npu2_tce_kill messages are from skiboot):
>
> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> ...
> pci 0004:04 : [PE# 00] Removing DMA window #0
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0004:04 : [PE# 00] Removing DMA window #1
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
>
> After the patch (no errors here):
>
> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
> pci 0004:04 : [PE# 00] Removing DMA window #0
> pci 0004:04 : [PE# 00] Removing DMA window #1
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
> ---
> arch/powerpc/platforms/powernv/pci-ioda.c | 27 ++++++++++++++++++++++++---
> 1 file changed, 24 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 496e476..2f91815 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -2681,14 +2681,23 @@ static struct pnv_ioda_pe *gpe_table_group_to_npe(
> static long pnv_pci_ioda2_npu_set_window(struct iommu_table_group *table_group,
> int num, struct iommu_table *tbl)
> {
> + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group);
> + int num2 = (num == 0) ? 1 : 0;
> long ret = pnv_pci_ioda2_set_window(table_group, num, tbl);
>
> if (ret)
> return ret;
>
> - ret = pnv_npu_set_window(gpe_table_group_to_npe(table_group), num, tbl);
> - if (ret)
> + if (table_group->tables[num2])
> + pnv_npu_unset_window(npe, num2);
> +
> + ret = pnv_npu_set_window(npe, num, tbl);
> + if (ret) {
> pnv_pci_ioda2_unset_window(table_group, num);
> + if (table_group->tables[num2])
> + pnv_npu_set_window(npe, num2,
> + table_group->tables[num2]);
> + }
>
> return ret;
> }
> @@ -2697,12 +2706,24 @@ static long pnv_pci_ioda2_npu_unset_window(
> struct iommu_table_group *table_group,
> int num)
> {
> + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group);
> + int num2 = (num == 0) ? 1 : 0;
> long ret = pnv_pci_ioda2_unset_window(table_group, num);
>
> if (ret)
> return ret;
>
> - return pnv_npu_unset_window(gpe_table_group_to_npe(table_group), num);
> + if (!npe->table_group.tables[num])
> + return 0;
> +
> + ret = pnv_npu_unset_window(npe, num);
> + if (ret)
> + return ret;
> +
> + if (table_group->tables[num2])
> + ret = pnv_npu_set_window(npe, num2, table_group->tables[num2]);
> +
> + return ret;
> }
>
> static void pnv_ioda2_npu_take_ownership(struct iommu_table_group *table_group)
>
--
Alexey
^ permalink raw reply
* Re: [PATCH 9/9] powerpc/eeh: Add eeh_state_active() helper
From: Sam Bobroff @ 2018-03-07 3:33 UTC (permalink / raw)
To: Russell Currey; +Cc: linuxppc-dev
In-Reply-To: <1520315388.2077.15.camel@russell.cc>
[-- Attachment #1: Type: text/plain, Size: 510 bytes --]
On Tue, Mar 06, 2018 at 04:49:48PM +1100, Russell Currey wrote:
> On Tue, 2018-03-06 at 11:00 +1100, Sam Bobroff wrote:
> > Checking for a "fully active" device state requires testing two flag
> > bits, which is open coded in several places, so add a function to do
> > it.
> >
> > Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
>
> Thanks for the patches.
Thanks for the review, I'll do a v2 with all the nits mentioned so far,
Sam.
> Reviewed-by: Russell Currey <ruscur@russell.cc>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [PATCH v2 10/10] powerpc/mm/slice: use the dynamic high slice size to limit bitmap operations
From: Nicholas Piggin @ 2018-03-07 3:22 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-11-npiggin@gmail.com>
On Wed, 7 Mar 2018 11:37:18 +1000
Nicholas Piggin <npiggin@gmail.com> wrote:
> The number of high slices a process might use now depends on its
> address space size, and what allocation address it has requested.
>
> This patch uses that limit throughout call chains where possible,
> rather than use the fixed SLICE_NUM_HIGH for bitmap operations.
> This saves some cost for processes that don't use very large address
> spaces.
>
> Perormance numbers aren't changed significantly, this may change
> with larger address spaces or different mmap access patterns that
> require more slice mask building.
Ignore this patch in the series. I didn't intend to send it.
Thanks,
Nick
^ permalink raw reply
* [PATCH v2 10/10] powerpc/mm/slice: use the dynamic high slice size to limit bitmap operations
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
The number of high slices a process might use now depends on its
address space size, and what allocation address it has requested.
This patch uses that limit throughout call chains where possible,
rather than use the fixed SLICE_NUM_HIGH for bitmap operations.
This saves some cost for processes that don't use very large address
spaces.
Perormance numbers aren't changed significantly, this may change
with larger address spaces or different mmap access patterns that
require more slice mask building.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 75 +++++++++++++++++++++++++++++--------------------
1 file changed, 45 insertions(+), 30 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 1297b3ad7dd2..09b95e976de9 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -61,14 +61,12 @@ static void slice_print_mask(const char *label, const struct slice_mask *mask) {
#endif
static void slice_range_to_mask(unsigned long start, unsigned long len,
- struct slice_mask *ret)
+ struct slice_mask *ret,
+ unsigned long high_slices)
{
unsigned long end = start + len - 1;
ret->low_slices = 0;
- if (SLICE_NUM_HIGH)
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
-
if (start < SLICE_LOW_TOP) {
unsigned long mend = min(end,
(unsigned long)(SLICE_LOW_TOP - 1));
@@ -77,6 +75,11 @@ static void slice_range_to_mask(unsigned long start, unsigned long len,
- (1u << GET_LOW_SLICE_INDEX(start));
}
+ if (!SLICE_NUM_HIGH)
+ return;
+
+#error XXX: bitmap_zero with non-const size? Good code?
+ bitmap_zero(ret->high_slices, high_slices);
if ((start + len) > SLICE_LOW_TOP) {
unsigned long start_index = GET_HIGH_SLICE_INDEX(start);
unsigned long align_end = ALIGN(end, (1UL << SLICE_HIGH_SHIFT));
@@ -120,22 +123,20 @@ static int slice_high_has_vma(struct mm_struct *mm, unsigned long slice)
}
static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
- unsigned long high_limit)
+ unsigned long high_slices)
{
unsigned long i;
ret->low_slices = 0;
- if (SLICE_NUM_HIGH)
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
-
for (i = 0; i < SLICE_NUM_LOW; i++)
if (!slice_low_has_vma(mm, i))
ret->low_slices |= 1u << i;
- if (high_limit <= SLICE_LOW_TOP)
+ if (!SLICE_NUM_HIGH || !high_slices)
return;
- for (i = 0; i < GET_HIGH_SLICE_INDEX(high_limit); i++)
+ bitmap_zero(ret->high_slices, high_slices);
+ for (i = 0; i < high_slices; i++)
if (!slice_high_has_vma(mm, i))
__set_bit(i, ret->high_slices);
}
@@ -228,6 +229,7 @@ static void slice_convert(struct mm_struct *mm,
{
int index, mask_index;
/* Write the new slice psize bits */
+ unsigned long high_slices;
unsigned char *hpsizes, *lpsizes;
struct slice_mask *psize_mask, *old_mask;
unsigned long i, flags;
@@ -263,7 +265,8 @@ static void slice_convert(struct mm_struct *mm,
}
hpsizes = mm->context.high_slices_psize;
- for (i = 0; i < GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit); i++) {
+ high_slices = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ for (i = 0; SLICE_NUM_HIGH && i < high_slices; i++) {
if (!test_bit(i, mask->high_slices))
continue;
@@ -430,32 +433,35 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
}
static inline void slice_copy_mask(struct slice_mask *dst,
- const struct slice_mask *src)
+ const struct slice_mask *src,
+ unsigned long high_slices)
{
dst->low_slices = src->low_slices;
if (!SLICE_NUM_HIGH)
return;
- bitmap_copy(dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
+ bitmap_copy(dst->high_slices, src->high_slices, high_slices);
}
static inline void slice_or_mask(struct slice_mask *dst,
const struct slice_mask *src1,
- const struct slice_mask *src2)
+ const struct slice_mask *src2,
+ unsigned long high_slices)
{
dst->low_slices = src1->low_slices | src2->low_slices;
if (!SLICE_NUM_HIGH)
return;
- bitmap_or(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
+ bitmap_or(dst->high_slices, src1->high_slices, src2->high_slices, high_slices);
}
static inline void slice_andnot_mask(struct slice_mask *dst,
const struct slice_mask *src1,
- const struct slice_mask *src2)
+ const struct slice_mask *src2,
+ unsigned long high_slices)
{
dst->low_slices = src1->low_slices & ~src2->low_slices;
if (!SLICE_NUM_HIGH)
return;
- bitmap_andnot(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
+ bitmap_andnot(dst->high_slices, src1->high_slices, src2->high_slices, high_slices);
}
#ifdef CONFIG_PPC_64K_PAGES
@@ -478,6 +484,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
struct mm_struct *mm = current->mm;
unsigned long newaddr;
unsigned long high_limit;
+ unsigned long high_slices;
high_limit = DEFAULT_MAP_WINDOW;
if (addr >= high_limit || (fixed && (addr + len > high_limit)))
@@ -494,6 +501,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
return -ENOMEM;
}
+ high_slices = GET_HIGH_SLICE_INDEX(high_limit);
if (high_limit > mm->context.slb_addr_limit) {
/*
* Increasing the slb_addr_limit does not require
@@ -556,11 +564,11 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (IS_ENABLED(CONFIG_PPC_64K_PAGES) && psize == MMU_PAGE_64K) {
compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
if (fixed)
- slice_or_mask(&good_mask, maskp, compat_maskp);
+ slice_or_mask(&good_mask, maskp, compat_maskp, high_slices);
else
- slice_copy_mask(&good_mask, maskp);
+ slice_copy_mask(&good_mask, maskp, high_slices);
} else {
- slice_copy_mask(&good_mask, maskp);
+ slice_copy_mask(&good_mask, maskp, high_slices);
}
slice_print_mask(" good_mask", &good_mask);
@@ -594,8 +602,8 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* We don't fit in the good mask, check what other slices are
* empty and thus can be converted
*/
- slice_mask_for_free(mm, &potential_mask, high_limit);
- slice_or_mask(&potential_mask, &potential_mask, &good_mask);
+ slice_mask_for_free(mm, &potential_mask, high_slices);
+ slice_or_mask(&potential_mask, &potential_mask, &good_mask, high_slices);
slice_print_mask(" potential", &potential_mask);
if (addr != 0 || fixed) {
@@ -632,7 +640,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
if (addr == -ENOMEM && psize == MMU_PAGE_64K) {
/* retry the search with 4k-page slices included */
- slice_or_mask(&potential_mask, &potential_mask, compat_maskp);
+ slice_or_mask(&potential_mask, &potential_mask, compat_maskp, high_slices);
addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
}
@@ -641,17 +649,18 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (addr == -ENOMEM)
return -ENOMEM;
- slice_range_to_mask(addr, len, &potential_mask);
+ slice_range_to_mask(addr, len, &potential_mask, high_slices);
slice_dbg(" found potential area at 0x%lx\n", addr);
slice_print_mask(" mask", &potential_mask);
convert:
- slice_andnot_mask(&potential_mask, &potential_mask, &good_mask);
+ slice_andnot_mask(&potential_mask, &potential_mask, &good_mask, high_slices);
if (compat_maskp && !fixed)
- slice_andnot_mask(&potential_mask, &potential_mask, compat_maskp);
+ slice_andnot_mask(&potential_mask, &potential_mask, compat_maskp, high_slices);
+#error XXX: are we sure high_slices is > 0 here when SLICE_NUM_HIGH? Perhaps not for 32-bit tasks on 64?
if (potential_mask.low_slices ||
(SLICE_NUM_HIGH &&
- !bitmap_empty(potential_mask.high_slices, SLICE_NUM_HIGH))) {
+ !bitmap_empty(potential_mask.high_slices, high_slices))) {
slice_convert(mm, &potential_mask, psize);
if (psize > MMU_PAGE_BASE)
on_each_cpu(slice_flush_segments, mm, 1);
@@ -722,7 +731,9 @@ void slice_init_new_context_exec(struct mm_struct *mm)
mm->context.user_psize = psize;
/*
- * Set all slice psizes to the default.
+ * Set all slice psizes to the default. High slices could
+ * be initialised up to slb_addr_limit if we ensure to
+ * initialise the rest of them as slb_addr_limit is expanded.
*/
lpsizes = mm->context.low_slices_psize;
memset(lpsizes, (psize << 4) | psize, SLICE_NUM_LOW >> 1);
@@ -743,10 +754,12 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
unsigned long len, unsigned int psize)
{
struct slice_mask mask;
+ unsigned long high_slices;
VM_BUG_ON(radix_enabled());
- slice_range_to_mask(start, len, &mask);
+ high_slices = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ slice_range_to_mask(start, len, &mask, high_slices);
slice_convert(mm, &mask, psize);
}
@@ -784,9 +797,11 @@ int slice_is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
if (psize == MMU_PAGE_64K) {
const struct slice_mask *compat_maskp;
struct slice_mask available;
+ unsigned long high_slices;
compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
- slice_or_mask(&available, maskp, compat_maskp);
+ high_slices = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ slice_or_mask(&available, maskp, compat_maskp, high_slices);
return !slice_check_range_fits(mm, &available, addr, len);
}
#endif
--
2.16.1
^ permalink raw reply related
* [PATCH v2 09/10] powerpc/mm/slice: remove radix calls to the slice code
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
This is a tidy up which removes radix MMU calls into the slice
code.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/hugetlb.h | 7 ++++---
arch/powerpc/mm/hugetlbpage.c | 6 ++++--
arch/powerpc/mm/slice.c | 17 ++++-------------
3 files changed, 12 insertions(+), 18 deletions(-)
diff --git a/arch/powerpc/include/asm/hugetlb.h b/arch/powerpc/include/asm/hugetlb.h
index 1a4847f67ea8..9e168407cd1e 100644
--- a/arch/powerpc/include/asm/hugetlb.h
+++ b/arch/powerpc/include/asm/hugetlb.h
@@ -90,16 +90,17 @@ pte_t *huge_pte_offset_and_shift(struct mm_struct *mm,
void flush_dcache_icache_hugepage(struct page *page);
#if defined(CONFIG_PPC_MM_SLICES)
-int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
+int slice_is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
unsigned long len);
-#else
+#endif
static inline int is_hugepage_only_range(struct mm_struct *mm,
unsigned long addr,
unsigned long len)
{
+ if (IS_ENABLED(CONFIG_PPC_MM_SLICES) && !radix_enabled())
+ return slice_is_hugepage_only_range(mm, addr, len);
return 0;
}
-#endif
void book3e_hugetlb_preload(struct vm_area_struct *vma, unsigned long ea,
pte_t pte);
diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
index 590be3fa0ce2..f4153f21d214 100644
--- a/arch/powerpc/mm/hugetlbpage.c
+++ b/arch/powerpc/mm/hugetlbpage.c
@@ -565,10 +565,12 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
unsigned long vma_mmu_pagesize(struct vm_area_struct *vma)
{
#ifdef CONFIG_PPC_MM_SLICES
- unsigned int psize = get_slice_psize(vma->vm_mm, vma->vm_start);
/* With radix we don't use slice, so derive it from vma*/
- if (!radix_enabled())
+ if (!radix_enabled()) {
+ unsigned int psize = get_slice_psize(vma->vm_mm, vma->vm_start);
+
return 1UL << mmu_psize_to_shift(psize);
+ }
#endif
if (!is_vm_hugetlb_page(vma))
return PAGE_SIZE;
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index b3b465c37224..1297b3ad7dd2 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -686,16 +686,8 @@ unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr)
unsigned char *psizes;
int index, mask_index;
- /*
- * Radix doesn't use slice, but can get enabled along with MMU_SLICE
- */
- if (radix_enabled()) {
-#ifdef CONFIG_PPC_64K_PAGES
- return MMU_PAGE_64K;
-#else
- return MMU_PAGE_4K;
-#endif
- }
+ VM_BUG_ON(radix_enabled());
+
if (addr < SLICE_LOW_TOP) {
psizes = mm->context.low_slices_psize;
index = GET_LOW_SLICE_INDEX(addr);
@@ -778,14 +770,13 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
* for now as we only use slices with hugetlbfs enabled. This should
* be fixed as the generic code gets fixed.
*/
-int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
+int slice_is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
unsigned long len)
{
const struct slice_mask *maskp;
unsigned int psize = mm->context.user_psize;
- if (radix_enabled())
- return 0;
+ VM_BUG_ON(radix_enabled());
maskp = slice_mask_for_size(mm, psize);
#ifdef CONFIG_PPC_64K_PAGES
--
2.16.1
^ permalink raw reply related
* [PATCH v2 08/10] powerpc/mm/slice: Use const pointers to cached slice masks where possible
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
The slice_mask cache was a basic conversion which copied the slice
mask into caller's structures, because that's how the original code
worked. In most cases the pointer can be used directly instead, saving
a copy and an on-stack structure.
On POWER8, this increases vfork+exec+exit performance by 0.3%
and reduces time to mmap+munmap a 64kB page by 2%.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 79 ++++++++++++++++++++++++-------------------------
1 file changed, 38 insertions(+), 41 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index c4cb4de1fab5..b3b465c37224 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -468,10 +468,10 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
unsigned long flags, unsigned int psize,
int topdown)
{
- struct slice_mask mask;
struct slice_mask good_mask;
struct slice_mask potential_mask;
- struct slice_mask compat_mask;
+ const struct slice_mask *maskp;
+ const struct slice_mask *compat_maskp = NULL;
int fixed = (flags & MAP_FIXED);
int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT);
unsigned long page_size = 1UL << pshift;
@@ -505,22 +505,6 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
on_each_cpu(slice_flush_segments, mm, 1);
}
- /*
- * init different masks
- */
- mask.low_slices = 0;
-
- /* silence stupid warning */;
- potential_mask.low_slices = 0;
-
- compat_mask.low_slices = 0;
-
- if (SLICE_NUM_HIGH) {
- bitmap_zero(mask.high_slices, SLICE_NUM_HIGH);
- bitmap_zero(potential_mask.high_slices, SLICE_NUM_HIGH);
- bitmap_zero(compat_mask.high_slices, SLICE_NUM_HIGH);
- }
-
/* Sanity checks */
BUG_ON(mm->task_size == 0);
BUG_ON(mm->context.slb_addr_limit == 0);
@@ -543,8 +527,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* First make up a "good" mask of slices that have the right size
* already
*/
- good_mask = *slice_mask_for_size(mm, psize);
- slice_print_mask(" good_mask", &good_mask);
+ maskp = slice_mask_for_size(mm, psize);
/*
* Here "good" means slices that are already the right page size,
@@ -565,14 +548,24 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* search in good | compat | free, found => convert free.
*/
-#ifdef CONFIG_PPC_64K_PAGES
- /* If we support combo pages, we can allow 64k pages in 4k slices */
- if (psize == MMU_PAGE_64K) {
- compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
+ /*
+ * If we support combo pages, we can allow 64k pages in 4k slices
+ * The mask copies could be avoided in most cases here if we had
+ * a pointer to good mask for the next code to use.
+ */
+ if (IS_ENABLED(CONFIG_PPC_64K_PAGES) && psize == MMU_PAGE_64K) {
+ compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
if (fixed)
- slice_or_mask(&good_mask, &good_mask, &compat_mask);
+ slice_or_mask(&good_mask, maskp, compat_maskp);
+ else
+ slice_copy_mask(&good_mask, maskp);
+ } else {
+ slice_copy_mask(&good_mask, maskp);
}
-#endif
+
+ slice_print_mask(" good_mask", &good_mask);
+ if (compat_maskp)
+ slice_print_mask(" compat_mask", compat_maskp);
/* First check hint if it's valid or if we have MAP_FIXED */
if (addr != 0 || fixed) {
@@ -639,7 +632,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
if (addr == -ENOMEM && psize == MMU_PAGE_64K) {
/* retry the search with 4k-page slices included */
- slice_or_mask(&potential_mask, &potential_mask, &compat_mask);
+ slice_or_mask(&potential_mask, &potential_mask, compat_maskp);
addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
}
@@ -648,17 +641,18 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (addr == -ENOMEM)
return -ENOMEM;
- slice_range_to_mask(addr, len, &mask);
+ slice_range_to_mask(addr, len, &potential_mask);
slice_dbg(" found potential area at 0x%lx\n", addr);
- slice_print_mask(" mask", &mask);
+ slice_print_mask(" mask", &potential_mask);
convert:
- slice_andnot_mask(&mask, &mask, &good_mask);
- slice_andnot_mask(&mask, &mask, &compat_mask);
- if (mask.low_slices ||
- (SLICE_NUM_HIGH &&
- !bitmap_empty(mask.high_slices, SLICE_NUM_HIGH))) {
- slice_convert(mm, &mask, psize);
+ slice_andnot_mask(&potential_mask, &potential_mask, &good_mask);
+ if (compat_maskp && !fixed)
+ slice_andnot_mask(&potential_mask, &potential_mask, compat_maskp);
+ if (potential_mask.low_slices ||
+ (SLICE_NUM_HIGH &&
+ !bitmap_empty(potential_mask.high_slices, SLICE_NUM_HIGH))) {
+ slice_convert(mm, &potential_mask, psize);
if (psize > MMU_PAGE_BASE)
on_each_cpu(slice_flush_segments, mm, 1);
}
@@ -787,22 +781,25 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
unsigned long len)
{
- struct slice_mask available;
+ const struct slice_mask *maskp;
unsigned int psize = mm->context.user_psize;
if (radix_enabled())
return 0;
- available = *slice_mask_for_size(mm, psize);
+ maskp = slice_mask_for_size(mm, psize);
#ifdef CONFIG_PPC_64K_PAGES
/* We need to account for 4k slices too */
if (psize == MMU_PAGE_64K) {
- struct slice_mask compat_mask;
- compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
- slice_or_mask(&available, &available, &compat_mask);
+ const struct slice_mask *compat_maskp;
+ struct slice_mask available;
+
+ compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
+ slice_or_mask(&available, maskp, compat_maskp);
+ return !slice_check_range_fits(mm, &available, addr, len);
}
#endif
- return !slice_check_range_fits(mm, &available, addr, len);
+ return !slice_check_range_fits(mm, maskp, addr, len);
}
#endif
--
2.16.1
^ permalink raw reply related
* [PATCH v2 07/10] powerpc/mm/slice: remove dead code
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
This code is never compiled in, and it gets broken by the next
patch, so remove it.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 6 ------
1 file changed, 6 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 4b2fd37b727a..c4cb4de1fab5 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -803,12 +803,6 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
}
#endif
-#if 0 /* too verbose */
- slice_dbg("is_hugepage_only_range(mm=%p, addr=%lx, len=%lx)\n",
- mm, addr, len);
- slice_print_mask(" mask", &mask);
- slice_print_mask(" available", &available);
-#endif
return !slice_check_range_fits(mm, &available, addr, len);
}
#endif
--
2.16.1
^ permalink raw reply related
* [PATCH v2 06/10] powerpc/mm/slice: Switch to 3-operand slice bitops helpers
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
This converts the slice_mask bit operation helpers to be the usual
3-operand kind, which allows 2 inputs to set a different output
without an extra copy, which is used in the next patch.
Adds slice_copy_mask, which will be used in the next patch.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 38 +++++++++++++++++++++++---------------
1 file changed, 23 insertions(+), 15 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 0a5efa40e739..4b2fd37b727a 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -429,25 +429,33 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
return slice_find_area_bottomup(mm, len, mask, psize, high_limit);
}
-static inline void slice_or_mask(struct slice_mask *dst,
+static inline void slice_copy_mask(struct slice_mask *dst,
const struct slice_mask *src)
{
- dst->low_slices |= src->low_slices;
+ dst->low_slices = src->low_slices;
if (!SLICE_NUM_HIGH)
return;
- bitmap_or(dst->high_slices, dst->high_slices, src->high_slices,
- SLICE_NUM_HIGH);
+ bitmap_copy(dst->high_slices, src->high_slices, SLICE_NUM_HIGH);
}
-static inline void slice_andnot_mask(struct slice_mask *dst,
- const struct slice_mask *src)
+static inline void slice_or_mask(struct slice_mask *dst,
+ const struct slice_mask *src1,
+ const struct slice_mask *src2)
{
- dst->low_slices &= ~src->low_slices;
+ dst->low_slices = src1->low_slices | src2->low_slices;
+ if (!SLICE_NUM_HIGH)
+ return;
+ bitmap_or(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
+}
+static inline void slice_andnot_mask(struct slice_mask *dst,
+ const struct slice_mask *src1,
+ const struct slice_mask *src2)
+{
+ dst->low_slices = src1->low_slices & ~src2->low_slices;
if (!SLICE_NUM_HIGH)
return;
- bitmap_andnot(dst->high_slices, dst->high_slices, src->high_slices,
- SLICE_NUM_HIGH);
+ bitmap_andnot(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
}
#ifdef CONFIG_PPC_64K_PAGES
@@ -562,7 +570,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (psize == MMU_PAGE_64K) {
compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
if (fixed)
- slice_or_mask(&good_mask, &compat_mask);
+ slice_or_mask(&good_mask, &good_mask, &compat_mask);
}
#endif
@@ -594,7 +602,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* empty and thus can be converted
*/
slice_mask_for_free(mm, &potential_mask, high_limit);
- slice_or_mask(&potential_mask, &good_mask);
+ slice_or_mask(&potential_mask, &potential_mask, &good_mask);
slice_print_mask(" potential", &potential_mask);
if (addr != 0 || fixed) {
@@ -631,7 +639,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
if (addr == -ENOMEM && psize == MMU_PAGE_64K) {
/* retry the search with 4k-page slices included */
- slice_or_mask(&potential_mask, &compat_mask);
+ slice_or_mask(&potential_mask, &potential_mask, &compat_mask);
addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
}
@@ -645,8 +653,8 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
slice_print_mask(" mask", &mask);
convert:
- slice_andnot_mask(&mask, &good_mask);
- slice_andnot_mask(&mask, &compat_mask);
+ slice_andnot_mask(&mask, &mask, &good_mask);
+ slice_andnot_mask(&mask, &mask, &compat_mask);
if (mask.low_slices ||
(SLICE_NUM_HIGH &&
!bitmap_empty(mask.high_slices, SLICE_NUM_HIGH))) {
@@ -791,7 +799,7 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
if (psize == MMU_PAGE_64K) {
struct slice_mask compat_mask;
compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
- slice_or_mask(&available, &compat_mask);
+ slice_or_mask(&available, &available, &compat_mask);
}
#endif
--
2.16.1
^ permalink raw reply related
* [PATCH v2 05/10] powerpc/mm/slice: implement slice_check_range_fits
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
Rather than build slice masks from a range then use that to check for
fit in a candidate mask, implement slice_check_range_fits that checks
if a range fits in a mask directly.
This allows several structures to be removed from stacks, and also we
don't expect a huge range in a lot of these cases, so building and
comparing a full mask is going to be more expensive than testing just
one or two bits of the range.
On POWER8, this increases vfork+exec+exit performance by 0.3%
and reduces time to mmap+munmap a 64kB page by 5%.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 62 +++++++++++++++++++++++++++----------------------
1 file changed, 34 insertions(+), 28 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 3e199b9cbbfd..0a5efa40e739 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -174,26 +174,36 @@ static struct slice_mask *slice_mask_for_size(struct mm_struct *mm, int psize)
#error "Must define the slice masks for page sizes supported by the platform"
#endif
-static int slice_check_fit(struct mm_struct *mm,
- const struct slice_mask *mask,
- const struct slice_mask *available)
+static bool slice_check_range_fits(struct mm_struct *mm,
+ const struct slice_mask *available,
+ unsigned long start, unsigned long len)
{
- DECLARE_BITMAP(result, SLICE_NUM_HIGH);
- /*
- * Make sure we just do bit compare only to the max
- * addr limit and not the full bit map size.
- */
- unsigned long slice_count = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
+ unsigned long end = start + len - 1;
+ u64 low_slices = 0;
- if (!SLICE_NUM_HIGH)
- return (mask->low_slices & available->low_slices) ==
- mask->low_slices;
+ if (start < SLICE_LOW_TOP) {
+ unsigned long mend = min(end,
+ (unsigned long)(SLICE_LOW_TOP - 1));
- bitmap_and(result, mask->high_slices,
- available->high_slices, slice_count);
+ low_slices = (1u << (GET_LOW_SLICE_INDEX(mend) + 1))
+ - (1u << GET_LOW_SLICE_INDEX(start));
+ }
+ if ((low_slices & available->low_slices) != low_slices)
+ return false;
- return (mask->low_slices & available->low_slices) == mask->low_slices &&
- bitmap_equal(result, mask->high_slices, slice_count);
+ if (SLICE_NUM_HIGH && ((start + len) > SLICE_LOW_TOP)) {
+ unsigned long start_index = GET_HIGH_SLICE_INDEX(start);
+ unsigned long align_end = ALIGN(end, (1UL << SLICE_HIGH_SHIFT));
+ unsigned long count = GET_HIGH_SLICE_INDEX(align_end) - start_index;
+ unsigned long i;
+
+ for (i = start_index; i < start_index + count; i++) {
+ if (!test_bit(i, available->high_slices))
+ return false;
+ }
+ }
+
+ return true;
}
static void slice_flush_segments(void *parm)
@@ -558,14 +568,10 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* First check hint if it's valid or if we have MAP_FIXED */
if (addr != 0 || fixed) {
- /* Build a mask for the requested range */
- slice_range_to_mask(addr, len, &mask);
- slice_print_mask(" mask", &mask);
-
/* Check if we fit in the good mask. If we do, we just return,
* nothing else to do
*/
- if (slice_check_fit(mm, &mask, &good_mask)) {
+ if (slice_check_range_fits(mm, &good_mask, addr, len)) {
slice_dbg(" fits good !\n");
return addr;
}
@@ -591,10 +597,11 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
slice_or_mask(&potential_mask, &good_mask);
slice_print_mask(" potential", &potential_mask);
- if ((addr != 0 || fixed) &&
- slice_check_fit(mm, &mask, &potential_mask)) {
- slice_dbg(" fits potential !\n");
- goto convert;
+ if (addr != 0 || fixed) {
+ if (slice_check_range_fits(mm, &potential_mask, addr, len)) {
+ slice_dbg(" fits potential !\n");
+ goto convert;
+ }
}
/* If we have MAP_FIXED and failed the above steps, then error out */
@@ -772,13 +779,12 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
unsigned long len)
{
- struct slice_mask mask, available;
+ struct slice_mask available;
unsigned int psize = mm->context.user_psize;
if (radix_enabled())
return 0;
- slice_range_to_mask(addr, len, &mask);
available = *slice_mask_for_size(mm, psize);
#ifdef CONFIG_PPC_64K_PAGES
/* We need to account for 4k slices too */
@@ -795,6 +801,6 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
slice_print_mask(" mask", &mask);
slice_print_mask(" available", &available);
#endif
- return !slice_check_fit(mm, &mask, &available);
+ return !slice_check_range_fits(mm, &available, addr, len);
}
#endif
--
2.16.1
^ permalink raw reply related
* [PATCH v2 04/10] powerpc/mm/slice: implement a slice mask cache
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev
Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy,
Benjamin Herrenschmidt, Anton Blanchard
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
Calculating the slice mask can become a signifcant overhead for
get_unmapped_area. This patch adds a struct slice_mask for
each page size in the mm_context, and keeps these in synch with
the slices psize arrays and slb_addr_limit.
On Book3S/64 this adds 288 bytes to the mm_context_t for the
slice mask caches.
On POWER8, this increases vfork+exec+exit performance by 9.9%
and reduces time to mmap+munmap a 64kB page by 28%.
Reduces time to mmap+munmap by about 10% on 8xx.
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Anton Blanchard <anton@samba.org>
---
arch/powerpc/include/asm/book3s/64/mmu.h | 18 +++++
arch/powerpc/include/asm/mmu-8xx.h | 10 +++
arch/powerpc/mm/slice.c | 112 +++++++++++++++++++------------
3 files changed, 98 insertions(+), 42 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
index bef6e39ed63a..777778579305 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu.h
@@ -80,6 +80,16 @@ struct spinlock;
/* Maximum possible number of NPUs in a system. */
#define NV_MAX_NPUS 8
+/*
+ * One bit per slice. We have lower slices which cover 256MB segments
+ * upto 4G range. That gets us 16 low slices. For the rest we track slices
+ * in 1TB size.
+ */
+struct slice_mask {
+ u64 low_slices;
+ DECLARE_BITMAP(high_slices, SLICE_NUM_HIGH);
+};
+
typedef struct {
mm_context_id_t id;
u16 user_psize; /* page size index */
@@ -95,6 +105,14 @@ typedef struct {
unsigned char low_slices_psize[BITS_PER_LONG / BITS_PER_BYTE];
unsigned char high_slices_psize[SLICE_ARRAY_SIZE];
unsigned long slb_addr_limit;
+# ifdef CONFIG_PPC_64K_PAGES
+ struct slice_mask mask_64k;
+# endif
+ struct slice_mask mask_4k;
+# ifdef CONFIG_HUGETLB_PAGE
+ struct slice_mask mask_16m;
+ struct slice_mask mask_16g;
+# endif
#else
u16 sllp; /* SLB page size encoding */
#endif
diff --git a/arch/powerpc/include/asm/mmu-8xx.h b/arch/powerpc/include/asm/mmu-8xx.h
index d3d7e79140c6..4f547752ae79 100644
--- a/arch/powerpc/include/asm/mmu-8xx.h
+++ b/arch/powerpc/include/asm/mmu-8xx.h
@@ -192,6 +192,11 @@
#endif
#ifndef __ASSEMBLY__
+struct slice_mask {
+ u64 low_slices;
+ DECLARE_BITMAP(high_slices, 0);
+};
+
typedef struct {
unsigned int id;
unsigned int active;
@@ -201,6 +206,11 @@ typedef struct {
unsigned char low_slices_psize[SLICE_ARRAY_SIZE];
unsigned char high_slices_psize[0];
unsigned long slb_addr_limit;
+ struct slice_mask mask_base_psize; /* 4k or 16k */
+# ifdef CONFIG_HUGETLB_PAGE
+ struct slice_mask mask_512k;
+ struct slice_mask mask_8m;
+# endif
#endif
} mm_context_t;
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 420d791f0e18..3e199b9cbbfd 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -37,15 +37,6 @@
#include <asm/hugetlb.h>
static DEFINE_SPINLOCK(slice_convert_lock);
-/*
- * One bit per slice. We have lower slices which cover 256MB segments
- * upto 4G range. That gets us 16 low slices. For the rest we track slices
- * in 1TB size.
- */
-struct slice_mask {
- u64 low_slices;
- DECLARE_BITMAP(high_slices, SLICE_NUM_HIGH);
-};
#ifdef DEBUG
int _slice_debug = 1;
@@ -149,36 +140,39 @@ static void slice_mask_for_free(struct mm_struct *mm, struct slice_mask *ret,
__set_bit(i, ret->high_slices);
}
-static void slice_mask_for_size(struct mm_struct *mm, int psize, struct slice_mask *ret,
- unsigned long high_limit)
+#ifdef CONFIG_PPC_BOOK3S_64
+static struct slice_mask *slice_mask_for_size(struct mm_struct *mm, int psize)
{
- unsigned char *hpsizes, *lpsizes;
- int index, mask_index;
- unsigned long i;
-
- ret->low_slices = 0;
- if (SLICE_NUM_HIGH)
- bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
-
- lpsizes = mm->context.low_slices_psize;
- for (i = 0; i < SLICE_NUM_LOW; i++) {
- mask_index = i & 0x1;
- index = i >> 1;
- if (((lpsizes[index] >> (mask_index * 4)) & 0xf) == psize)
- ret->low_slices |= 1u << i;
- }
-
- if (high_limit <= SLICE_LOW_TOP)
- return;
-
- hpsizes = mm->context.high_slices_psize;
- for (i = 0; i < GET_HIGH_SLICE_INDEX(high_limit); i++) {
- mask_index = i & 0x1;
- index = i >> 1;
- if (((hpsizes[index] >> (mask_index * 4)) & 0xf) == psize)
- __set_bit(i, ret->high_slices);
- }
+#ifdef CONFIG_PPC_64K_PAGES
+ if (psize == MMU_PAGE_64K)
+ return &mm->context.mask_64k;
+#endif
+ if (psize == MMU_PAGE_4K)
+ return &mm->context.mask_4k;
+#ifdef CONFIG_HUGETLB_PAGE
+ if (psize == MMU_PAGE_16M)
+ return &mm->context.mask_16m;
+ if (psize == MMU_PAGE_16G)
+ return &mm->context.mask_16g;
+#endif
+ BUG();
}
+#elif defined(CONFIG_PPC_8xx)
+static struct slice_mask *slice_mask_for_size(struct mm_struct *mm, int psize)
+{
+ if (psize == mmu_virtual_psize)
+ return &mm->context.mask_base_psize;
+#ifdef CONFIG_HUGETLB_PAGE
+ if (psize == MMU_PAGE_512K)
+ return &mm->context.mask_512k;
+ if (psize == MMU_PAGE_8M)
+ return &mm->context.mask_8m;
+#endif
+ BUG();
+}
+#else
+#error "Must define the slice masks for page sizes supported by the platform"
+#endif
static int slice_check_fit(struct mm_struct *mm,
const struct slice_mask *mask,
@@ -225,11 +219,15 @@ static void slice_convert(struct mm_struct *mm,
int index, mask_index;
/* Write the new slice psize bits */
unsigned char *hpsizes, *lpsizes;
+ struct slice_mask *psize_mask, *old_mask;
unsigned long i, flags;
+ int old_psize;
slice_dbg("slice_convert(mm=%p, psize=%d)\n", mm, psize);
slice_print_mask(" mask", mask);
+ psize_mask = slice_mask_for_size(mm, psize);
+
/* We need to use a spinlock here to protect against
* concurrent 64k -> 4k demotion ...
*/
@@ -242,6 +240,14 @@ static void slice_convert(struct mm_struct *mm,
mask_index = i & 0x1;
index = i >> 1;
+
+ /* Update the slice_mask */
+ old_psize = (lpsizes[index] >> (mask_index * 4)) & 0xf;
+ old_mask = slice_mask_for_size(mm, old_psize);
+ old_mask->low_slices &= ~(1u << i);
+ psize_mask->low_slices |= 1u << i;
+
+ /* Update the sizes array */
lpsizes[index] = (lpsizes[index] & ~(0xf << (mask_index * 4))) |
(((unsigned long)psize) << (mask_index * 4));
}
@@ -253,6 +259,14 @@ static void slice_convert(struct mm_struct *mm,
mask_index = i & 0x1;
index = i >> 1;
+
+ /* Update the slice_mask */
+ old_psize = (hpsizes[index] >> (mask_index * 4)) & 0xf;
+ old_mask = slice_mask_for_size(mm, old_psize);
+ __clear_bit(i, old_mask->high_slices);
+ __set_bit(i, psize_mask->high_slices);
+
+ /* Update the sizes array */
hpsizes[index] = (hpsizes[index] & ~(0xf << (mask_index * 4))) |
(((unsigned long)psize) << (mask_index * 4));
}
@@ -463,7 +477,13 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
}
if (high_limit > mm->context.slb_addr_limit) {
+ /*
+ * Increasing the slb_addr_limit does not require
+ * slice mask cache to be recalculated because it should
+ * be already initialised beyond the old address limit.
+ */
mm->context.slb_addr_limit = high_limit;
+
on_each_cpu(slice_flush_segments, mm, 1);
}
@@ -505,7 +525,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* First make up a "good" mask of slices that have the right size
* already
*/
- slice_mask_for_size(mm, psize, &good_mask, high_limit);
+ good_mask = *slice_mask_for_size(mm, psize);
slice_print_mask(" good_mask", &good_mask);
/*
@@ -530,7 +550,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
#ifdef CONFIG_PPC_64K_PAGES
/* If we support combo pages, we can allow 64k pages in 4k slices */
if (psize == MMU_PAGE_64K) {
- slice_mask_for_size(mm, MMU_PAGE_4K, &compat_mask, high_limit);
+ compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
if (fixed)
slice_or_mask(&good_mask, &compat_mask);
}
@@ -682,6 +702,7 @@ EXPORT_SYMBOL_GPL(get_slice_psize);
void slice_init_new_context_exec(struct mm_struct *mm)
{
unsigned char *hpsizes, *lpsizes;
+ struct slice_mask *mask;
unsigned int psize = mmu_virtual_psize;
slice_dbg("slice_init_new_context_exec(mm=%p)\n", mm);
@@ -707,6 +728,14 @@ void slice_init_new_context_exec(struct mm_struct *mm)
hpsizes = mm->context.high_slices_psize;
memset(hpsizes, (psize << 4) | psize, SLICE_NUM_HIGH >> 1);
+
+ /*
+ * Slice mask cache starts zeroed, fill the default size cache.
+ */
+ mask = slice_mask_for_size(mm, psize);
+ mask->low_slices = ~0UL;
+ if (SLICE_NUM_HIGH)
+ bitmap_fill(mask->high_slices, SLICE_NUM_HIGH);
}
void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
@@ -745,18 +774,17 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
{
struct slice_mask mask, available;
unsigned int psize = mm->context.user_psize;
- unsigned long high_limit = mm->context.slb_addr_limit;
if (radix_enabled())
return 0;
slice_range_to_mask(addr, len, &mask);
- slice_mask_for_size(mm, psize, &available, high_limit);
+ available = *slice_mask_for_size(mm, psize);
#ifdef CONFIG_PPC_64K_PAGES
/* We need to account for 4k slices too */
if (psize == MMU_PAGE_64K) {
struct slice_mask compat_mask;
- slice_mask_for_size(mm, MMU_PAGE_4K, &compat_mask, high_limit);
+ compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
slice_or_mask(&available, &compat_mask);
}
#endif
--
2.16.1
^ permalink raw reply related
* [PATCH v2 03/10] powerpc/mm/slice: pass pointers to struct slice_mask where possible
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
Pass around const pointers to struct slice_mask where possible, rather
than copies of slice_mask, to reduce stack and call overhead.
checkstack.pl gives, before:
0x00000d1c slice_get_unmapped_area [slice.o]: 592
0x00001864 is_hugepage_only_range [slice.o]: 448
0x00000754 slice_find_area_topdown [slice.o]: 400
0x00000484 slice_find_area_bottomup.isra.1 [slice.o]: 272
0x000017b4 slice_set_range_psize [slice.o]: 224
0x00000a4c slice_find_area [slice.o]: 128
0x00000160 slice_check_fit [slice.o]: 112
after:
0x00000ad0 slice_get_unmapped_area [slice.o]: 448
0x00001464 is_hugepage_only_range [slice.o]: 288
0x000006c0 slice_find_area [slice.o]: 144
0x0000016c slice_check_fit [slice.o]: 128
0x00000528 slice_find_area_bottomup.isra.2 [slice.o]: 128
0x000013e4 slice_set_range_psize [slice.o]: 128
This increases vfork+exec+exit performance by 1.5%.
Reduces time to mmap+munmap a 64kB page by 17%.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 84 ++++++++++++++++++++++++++-----------------------
1 file changed, 45 insertions(+), 39 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 432c328b3e94..420d791f0e18 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -50,19 +50,21 @@ struct slice_mask {
#ifdef DEBUG
int _slice_debug = 1;
-static void slice_print_mask(const char *label, struct slice_mask mask)
+static void slice_print_mask(const char *label, const struct slice_mask *mask)
{
if (!_slice_debug)
return;
- pr_devel("%s low_slice: %*pbl\n", label, (int)SLICE_NUM_LOW, &mask.low_slices);
- pr_devel("%s high_slice: %*pbl\n", label, (int)SLICE_NUM_HIGH, mask.high_slices);
+ pr_devel("%s low_slice: %*pbl\n", label,
+ (int)SLICE_NUM_LOW, &mask->low_slices);
+ pr_devel("%s high_slice: %*pbl\n", label,
+ (int)SLICE_NUM_HIGH, mask->high_slices);
}
#define slice_dbg(fmt...) do { if (_slice_debug) pr_devel(fmt); } while (0)
#else
-static void slice_print_mask(const char *label, struct slice_mask mask) {}
+static void slice_print_mask(const char *label, const struct slice_mask *mask) {}
#define slice_dbg(fmt...)
#endif
@@ -179,7 +181,8 @@ static void slice_mask_for_size(struct mm_struct *mm, int psize, struct slice_ma
}
static int slice_check_fit(struct mm_struct *mm,
- struct slice_mask mask, struct slice_mask available)
+ const struct slice_mask *mask,
+ const struct slice_mask *available)
{
DECLARE_BITMAP(result, SLICE_NUM_HIGH);
/*
@@ -189,14 +192,14 @@ static int slice_check_fit(struct mm_struct *mm,
unsigned long slice_count = GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit);
if (!SLICE_NUM_HIGH)
- return (mask.low_slices & available.low_slices) ==
- mask.low_slices;
+ return (mask->low_slices & available->low_slices) ==
+ mask->low_slices;
- bitmap_and(result, mask.high_slices,
- available.high_slices, slice_count);
+ bitmap_and(result, mask->high_slices,
+ available->high_slices, slice_count);
- return (mask.low_slices & available.low_slices) == mask.low_slices &&
- bitmap_equal(result, mask.high_slices, slice_count);
+ return (mask->low_slices & available->low_slices) == mask->low_slices &&
+ bitmap_equal(result, mask->high_slices, slice_count);
}
static void slice_flush_segments(void *parm)
@@ -216,7 +219,8 @@ static void slice_flush_segments(void *parm)
#endif
}
-static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psize)
+static void slice_convert(struct mm_struct *mm,
+ const struct slice_mask *mask, int psize)
{
int index, mask_index;
/* Write the new slice psize bits */
@@ -233,7 +237,7 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
lpsizes = mm->context.low_slices_psize;
for (i = 0; i < SLICE_NUM_LOW; i++) {
- if (!(mask.low_slices & (1u << i)))
+ if (!(mask->low_slices & (1u << i)))
continue;
mask_index = i & 0x1;
@@ -244,7 +248,7 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
hpsizes = mm->context.high_slices_psize;
for (i = 0; i < GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit); i++) {
- if (!test_bit(i, mask.high_slices))
+ if (!test_bit(i, mask->high_slices))
continue;
mask_index = i & 0x1;
@@ -270,26 +274,25 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
* 'available' slice_mark.
*/
static bool slice_scan_available(unsigned long addr,
- struct slice_mask available,
- int end,
- unsigned long *boundary_addr)
+ const struct slice_mask *available,
+ int end, unsigned long *boundary_addr)
{
unsigned long slice;
if (addr < SLICE_LOW_TOP) {
slice = GET_LOW_SLICE_INDEX(addr);
*boundary_addr = (slice + end) << SLICE_LOW_SHIFT;
- return !!(available.low_slices & (1u << slice));
+ return !!(available->low_slices & (1u << slice));
} else {
slice = GET_HIGH_SLICE_INDEX(addr);
*boundary_addr = (slice + end) ?
((slice + end) << SLICE_HIGH_SHIFT) : SLICE_LOW_TOP;
- return !!test_bit(slice, available.high_slices);
+ return !!test_bit(slice, available->high_slices);
}
}
static unsigned long slice_find_area_bottomup(struct mm_struct *mm,
unsigned long len,
- struct slice_mask available,
+ const struct slice_mask *available,
int psize, unsigned long high_limit)
{
int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT);
@@ -335,7 +338,7 @@ static unsigned long slice_find_area_bottomup(struct mm_struct *mm,
static unsigned long slice_find_area_topdown(struct mm_struct *mm,
unsigned long len,
- struct slice_mask available,
+ const struct slice_mask *available,
int psize, unsigned long high_limit)
{
int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT);
@@ -393,7 +396,7 @@ static unsigned long slice_find_area_topdown(struct mm_struct *mm,
static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
- struct slice_mask mask, int psize,
+ const struct slice_mask *mask, int psize,
int topdown, unsigned long high_limit)
{
if (topdown)
@@ -402,7 +405,8 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
return slice_find_area_bottomup(mm, len, mask, psize, high_limit);
}
-static inline void slice_or_mask(struct slice_mask *dst, struct slice_mask *src)
+static inline void slice_or_mask(struct slice_mask *dst,
+ const struct slice_mask *src)
{
dst->low_slices |= src->low_slices;
if (!SLICE_NUM_HIGH)
@@ -411,7 +415,8 @@ static inline void slice_or_mask(struct slice_mask *dst, struct slice_mask *src)
SLICE_NUM_HIGH);
}
-static inline void slice_andnot_mask(struct slice_mask *dst, struct slice_mask *src)
+static inline void slice_andnot_mask(struct slice_mask *dst,
+ const struct slice_mask *src)
{
dst->low_slices &= ~src->low_slices;
@@ -501,7 +506,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* already
*/
slice_mask_for_size(mm, psize, &good_mask, high_limit);
- slice_print_mask(" good_mask", good_mask);
+ slice_print_mask(" good_mask", &good_mask);
/*
* Here "good" means slices that are already the right page size,
@@ -535,12 +540,12 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (addr != 0 || fixed) {
/* Build a mask for the requested range */
slice_range_to_mask(addr, len, &mask);
- slice_print_mask(" mask", mask);
+ slice_print_mask(" mask", &mask);
/* Check if we fit in the good mask. If we do, we just return,
* nothing else to do
*/
- if (slice_check_fit(mm, mask, good_mask)) {
+ if (slice_check_fit(mm, &mask, &good_mask)) {
slice_dbg(" fits good !\n");
return addr;
}
@@ -548,7 +553,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* Now let's see if we can find something in the existing
* slices for that size
*/
- newaddr = slice_find_area(mm, len, good_mask,
+ newaddr = slice_find_area(mm, len, &good_mask,
psize, topdown, high_limit);
if (newaddr != -ENOMEM) {
/* Found within the good mask, we don't have to setup,
@@ -564,9 +569,10 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
*/
slice_mask_for_free(mm, &potential_mask, high_limit);
slice_or_mask(&potential_mask, &good_mask);
- slice_print_mask(" potential", potential_mask);
+ slice_print_mask(" potential", &potential_mask);
- if ((addr != 0 || fixed) && slice_check_fit(mm, mask, potential_mask)) {
+ if ((addr != 0 || fixed) &&
+ slice_check_fit(mm, &mask, &potential_mask)) {
slice_dbg(" fits potential !\n");
goto convert;
}
@@ -581,7 +587,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* anywhere in the good area.
*/
if (addr) {
- addr = slice_find_area(mm, len, good_mask,
+ addr = slice_find_area(mm, len, &good_mask,
psize, topdown, high_limit);
if (addr != -ENOMEM) {
slice_dbg(" found area at 0x%lx\n", addr);
@@ -592,14 +598,14 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
/* Now let's see if we can find something in the existing slices
* for that size plus free slices
*/
- addr = slice_find_area(mm, len, potential_mask,
+ addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
#ifdef CONFIG_PPC_64K_PAGES
if (addr == -ENOMEM && psize == MMU_PAGE_64K) {
/* retry the search with 4k-page slices included */
slice_or_mask(&potential_mask, &compat_mask);
- addr = slice_find_area(mm, len, potential_mask,
+ addr = slice_find_area(mm, len, &potential_mask,
psize, topdown, high_limit);
}
#endif
@@ -609,7 +615,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
slice_range_to_mask(addr, len, &mask);
slice_dbg(" found potential area at 0x%lx\n", addr);
- slice_print_mask(" mask", mask);
+ slice_print_mask(" mask", &mask);
convert:
slice_andnot_mask(&mask, &good_mask);
@@ -617,7 +623,7 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
if (mask.low_slices ||
(SLICE_NUM_HIGH &&
!bitmap_empty(mask.high_slices, SLICE_NUM_HIGH))) {
- slice_convert(mm, mask, psize);
+ slice_convert(mm, &mask, psize);
if (psize > MMU_PAGE_BASE)
on_each_cpu(slice_flush_segments, mm, 1);
}
@@ -711,7 +717,7 @@ void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
VM_BUG_ON(radix_enabled());
slice_range_to_mask(start, len, &mask);
- slice_convert(mm, mask, psize);
+ slice_convert(mm, &mask, psize);
}
#ifdef CONFIG_HUGETLB_PAGE
@@ -758,9 +764,9 @@ int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
#if 0 /* too verbose */
slice_dbg("is_hugepage_only_range(mm=%p, addr=%lx, len=%lx)\n",
mm, addr, len);
- slice_print_mask(" mask", mask);
- slice_print_mask(" available", available);
+ slice_print_mask(" mask", &mask);
+ slice_print_mask(" available", &available);
#endif
- return !slice_check_fit(mm, mask, available);
+ return !slice_check_fit(mm, &mask, &available);
}
#endif
--
2.16.1
^ permalink raw reply related
* [PATCH v2 02/10] powerpc/mm/slice: tidy lpsizes and hpsizes update loops
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
Make these loops look the same, and change their form so the
important part is not wrapped over so many lines.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/slice.c | 22 ++++++++++++----------
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 7b51f962ce0c..432c328b3e94 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -232,22 +232,24 @@ static void slice_convert(struct mm_struct *mm, struct slice_mask mask, int psiz
spin_lock_irqsave(&slice_convert_lock, flags);
lpsizes = mm->context.low_slices_psize;
- for (i = 0; i < SLICE_NUM_LOW; i++)
- if (mask.low_slices & (1u << i)) {
- mask_index = i & 0x1;
- index = i >> 1;
- lpsizes[index] = (lpsizes[index] &
- ~(0xf << (mask_index * 4))) |
+ for (i = 0; i < SLICE_NUM_LOW; i++) {
+ if (!(mask.low_slices & (1u << i)))
+ continue;
+
+ mask_index = i & 0x1;
+ index = i >> 1;
+ lpsizes[index] = (lpsizes[index] & ~(0xf << (mask_index * 4))) |
(((unsigned long)psize) << (mask_index * 4));
- }
+ }
hpsizes = mm->context.high_slices_psize;
for (i = 0; i < GET_HIGH_SLICE_INDEX(mm->context.slb_addr_limit); i++) {
+ if (!test_bit(i, mask.high_slices))
+ continue;
+
mask_index = i & 0x1;
index = i >> 1;
- if (test_bit(i, mask.high_slices))
- hpsizes[index] = (hpsizes[index] &
- ~(0xf << (mask_index * 4))) |
+ hpsizes[index] = (hpsizes[index] & ~(0xf << (mask_index * 4))) |
(((unsigned long)psize) << (mask_index * 4));
}
--
2.16.1
^ permalink raw reply related
* [PATCH v2 01/10] powerpc/mm/slice: Simplify and optimise slice context initialisation
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
In-Reply-To: <20180307013718.13749-1-npiggin@gmail.com>
The slice state of an mm gets zeroed then initialised upon exec.
This is the only caller of slice_set_user_psize now, so that can be
removed and instead implement a faster and simplified approach that
requires no locking or checking existing state.
This speeds up vfork+exec+exit performance on POWER8 by 3%.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/slice.h | 8 ++--
arch/powerpc/mm/mmu_context_book3s64.c | 9 +----
arch/powerpc/mm/mmu_context_nohash.c | 5 +--
arch/powerpc/mm/slice.c | 72 +++++++++-------------------------
4 files changed, 23 insertions(+), 71 deletions(-)
diff --git a/arch/powerpc/include/asm/slice.h b/arch/powerpc/include/asm/slice.h
index 172711fadb1c..e40406cf5628 100644
--- a/arch/powerpc/include/asm/slice.h
+++ b/arch/powerpc/include/asm/slice.h
@@ -28,15 +28,13 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr);
-void slice_set_user_psize(struct mm_struct *mm, unsigned int psize);
void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
unsigned long len, unsigned int psize);
-#endif /* __ASSEMBLY__ */
-#else /* CONFIG_PPC_MM_SLICES */
+void slice_init_new_context_exec(struct mm_struct *mm);
+
+#endif /* __ASSEMBLY__ */
-#define slice_set_range_psize(mm, start, len, psize) \
- slice_set_user_psize((mm), (psize))
#endif /* CONFIG_PPC_MM_SLICES */
#endif /* _ASM_POWERPC_SLICE_H */
diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c
index 929d9ef7083f..80acad52b006 100644
--- a/arch/powerpc/mm/mmu_context_book3s64.c
+++ b/arch/powerpc/mm/mmu_context_book3s64.c
@@ -93,13 +93,6 @@ static int hash__init_new_context(struct mm_struct *mm)
if (index < 0)
return index;
- /*
- * In the case of exec, use the default limit,
- * otherwise inherit it from the mm we are duplicating.
- */
- if (!mm->context.slb_addr_limit)
- mm->context.slb_addr_limit = DEFAULT_MAP_WINDOW_USER64;
-
/*
* The old code would re-promote on fork, we don't do that when using
* slices as it could cause problem promoting slices that have been
@@ -115,7 +108,7 @@ static int hash__init_new_context(struct mm_struct *mm)
* check against 0 is OK.
*/
if (mm->context.id == 0)
- slice_set_user_psize(mm, mmu_virtual_psize);
+ slice_init_new_context_exec(mm);
subpage_prot_init_new_context(mm);
diff --git a/arch/powerpc/mm/mmu_context_nohash.c b/arch/powerpc/mm/mmu_context_nohash.c
index d98f7e5c141b..be8f5c9d4d08 100644
--- a/arch/powerpc/mm/mmu_context_nohash.c
+++ b/arch/powerpc/mm/mmu_context_nohash.c
@@ -332,9 +332,6 @@ int init_new_context(struct task_struct *t, struct mm_struct *mm)
pr_hard("initing context for mm @%p\n", mm);
#ifdef CONFIG_PPC_MM_SLICES
- if (!mm->context.slb_addr_limit)
- mm->context.slb_addr_limit = DEFAULT_MAP_WINDOW;
-
/*
* We have MMU_NO_CONTEXT set to be ~0. Hence check
* explicitly against context.id == 0. This ensures that we properly
@@ -343,7 +340,7 @@ int init_new_context(struct task_struct *t, struct mm_struct *mm)
* will have id != 0).
*/
if (mm->context.id == 0)
- slice_set_user_psize(mm, mmu_virtual_psize);
+ slice_init_new_context_exec(mm);
#endif
mm->context.id = MMU_NO_CONTEXT;
mm->context.active = 0;
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 5e9e1e57d580..7b51f962ce0c 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -671,70 +671,34 @@ unsigned int get_slice_psize(struct mm_struct *mm, unsigned long addr)
}
EXPORT_SYMBOL_GPL(get_slice_psize);
-/*
- * This is called by hash_page when it needs to do a lazy conversion of
- * an address space from real 64K pages to combo 4K pages (typically
- * when hitting a non cacheable mapping on a processor or hypervisor
- * that won't allow them for 64K pages).
- *
- * This is also called in init_new_context() to change back the user
- * psize from whatever the parent context had it set to
- * N.B. This may be called before mm->context.id has been set.
- *
- * This function will only change the content of the {low,high)_slice_psize
- * masks, it will not flush SLBs as this shall be handled lazily by the
- * caller.
- */
-void slice_set_user_psize(struct mm_struct *mm, unsigned int psize)
+void slice_init_new_context_exec(struct mm_struct *mm)
{
- int index, mask_index;
unsigned char *hpsizes, *lpsizes;
- unsigned long flags;
- unsigned int old_psize;
- int i;
+ unsigned int psize = mmu_virtual_psize;
- slice_dbg("slice_set_user_psize(mm=%p, psize=%d)\n", mm, psize);
+ slice_dbg("slice_init_new_context_exec(mm=%p)\n", mm);
- VM_BUG_ON(radix_enabled());
- spin_lock_irqsave(&slice_convert_lock, flags);
-
- old_psize = mm->context.user_psize;
- slice_dbg(" old_psize=%d\n", old_psize);
- if (old_psize == psize)
- goto bail;
+ /*
+ * In the case of exec, use the default limit. In the
+ * case of fork it is just inherited from the mm being
+ * duplicated.
+ */
+#ifdef CONFIG_PPC64
+ mm->context.slb_addr_limit = DEFAULT_MAP_WINDOW_USER64;
+#else
+ mm->context.slb_addr_limit = DEFAULT_MAP_WINDOW;
+#endif
mm->context.user_psize = psize;
- wmb();
+ /*
+ * Set all slice psizes to the default.
+ */
lpsizes = mm->context.low_slices_psize;
- for (i = 0; i < SLICE_NUM_LOW; i++) {
- mask_index = i & 0x1;
- index = i >> 1;
- if (((lpsizes[index] >> (mask_index * 4)) & 0xf) == old_psize)
- lpsizes[index] = (lpsizes[index] &
- ~(0xf << (mask_index * 4))) |
- (((unsigned long)psize) << (mask_index * 4));
- }
+ memset(lpsizes, (psize << 4) | psize, SLICE_NUM_LOW >> 1);
hpsizes = mm->context.high_slices_psize;
- for (i = 0; i < SLICE_NUM_HIGH; i++) {
- mask_index = i & 0x1;
- index = i >> 1;
- if (((hpsizes[index] >> (mask_index * 4)) & 0xf) == old_psize)
- hpsizes[index] = (hpsizes[index] &
- ~(0xf << (mask_index * 4))) |
- (((unsigned long)psize) << (mask_index * 4));
- }
-
-
-
-
- slice_dbg(" lsps=%lx, hsps=%lx\n",
- (unsigned long)mm->context.low_slices_psize,
- (unsigned long)mm->context.high_slices_psize);
-
- bail:
- spin_unlock_irqrestore(&slice_convert_lock, flags);
+ memset(hpsizes, (psize << 4) | psize, SLICE_NUM_HIGH >> 1);
}
void slice_set_range_psize(struct mm_struct *mm, unsigned long start,
--
2.16.1
^ permalink raw reply related
* [PATCH v2 00/10] powerpc/mm/slice: improve slice speed and stack use
From: Nicholas Piggin @ 2018-03-07 1:37 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V, Christophe Leroy
Overall on POWER8, this series increases vfork+exec+exit
microbenchmark rate by 15.6%, and mmap+munmap rate by 81%. Slice
code/data size is reduced by 1kB, and max stack overhead through
slice_get_unmapped_area call goes rom 992 to 448 bytes. The cost is
288 bytes added to the mm_context_t per mm for the slice masks on
Book3S.
Since v1:
- Fixed a couple of bugs and compile errors on 8xx.
- Accounted for all Christophe's review feedback hopefully.
- Got rid of unrelated "cleanup" hunks, and split one to its own patch.
- Dropped patch to dynamically limit bitmap operations. This may be
revisited after Aneesh's 4TB patches.
Thanks,
Nick
Nicholas Piggin (10):
powerpc/mm/slice: Simplify and optimise slice context initialisation
powerpc/mm/slice: tidy lpsizes and hpsizes update loops
powerpc/mm/slice: pass pointers to struct slice_mask where possible
powerpc/mm/slice: implement a slice mask cache
powerpc/mm/slice: implement slice_check_range_fits
powerpc/mm/slice: Switch to 3-operand slice bitops helpers
powerpc/mm/slice: remove dead code
powerpc/mm/slice: Use const pointers to cached slice masks where
possible
powerpc/mm/slice: remove radix calls to the slice code
powerpc/mm/slice: use the dynamic high slice size to limit bitmap
operations
arch/powerpc/include/asm/book3s/64/mmu.h | 18 ++
arch/powerpc/include/asm/hugetlb.h | 7 +-
arch/powerpc/include/asm/mmu-8xx.h | 10 +
arch/powerpc/include/asm/slice.h | 8 +-
arch/powerpc/mm/hugetlbpage.c | 6 +-
arch/powerpc/mm/mmu_context_book3s64.c | 9 +-
arch/powerpc/mm/mmu_context_nohash.c | 5 +-
arch/powerpc/mm/slice.c | 461 ++++++++++++++++---------------
8 files changed, 277 insertions(+), 247 deletions(-)
--
2.16.1
^ permalink raw reply
* Re: [PATCH 10/10] powerpc/mm/slice: remove radix calls to the slice code
From: Nicholas Piggin @ 2018-03-06 23:35 UTC (permalink / raw)
To: Christophe LEROY; +Cc: linuxppc-dev, Aneesh Kumar K . V
In-Reply-To: <01d94f39-91fc-53c4-79ad-b43b3234c2ea@c-s.fr>
On Tue, 6 Mar 2018 16:12:34 +0100
Christophe LEROY <christophe.leroy@c-s.fr> wrote:
> Le 06/03/2018 à 14:25, Nicholas Piggin a écrit :
> > This is a tidy up which removes radix MMU calls into the slice
> > code.
> >
> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> > ---
> > arch/powerpc/include/asm/hugetlb.h | 9 ++++++---
> > arch/powerpc/mm/hugetlbpage.c | 5 +++--
> > arch/powerpc/mm/slice.c | 17 ++++-------------
> > 3 files changed, 13 insertions(+), 18 deletions(-)
> >
> > diff --git a/arch/powerpc/include/asm/hugetlb.h b/arch/powerpc/include/asm/hugetlb.h
> > index 1a4847f67ea8..59885d444695 100644
> > --- a/arch/powerpc/include/asm/hugetlb.h
> > +++ b/arch/powerpc/include/asm/hugetlb.h
> > @@ -90,16 +90,19 @@ pte_t *huge_pte_offset_and_shift(struct mm_struct *mm,
> > void flush_dcache_icache_hugepage(struct page *page);
> >
> > #if defined(CONFIG_PPC_MM_SLICES)
> > -int is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
> > +int slice_is_hugepage_only_range(struct mm_struct *mm, unsigned long addr,
> > unsigned long len);
> > -#else
> > +#endif
> > static inline int is_hugepage_only_range(struct mm_struct *mm,
> > unsigned long addr,
> > unsigned long len)
> > {
> > +#if defined(CONFIG_PPC_MM_SLICES)
> > + if (!radix_enabled())
> > + return slice_is_hugepage_only_range(mm, addr, len);
> > +#endif
> > return 0;
>
> Might be easier to understand as
>
> if (!IS_ENABLED(CONFIG_PPC_MM_SLICES) || radix_enabled())
> return 0;
> return slice_is_hugepage_only_range(mm, addr, len);
Yep.
> > unsigned long vma_mmu_pagesize(struct vm_area_struct *vma)
> > {
> > #ifdef CONFIG_PPC_MM_SLICES
> > - unsigned int psize = get_slice_psize(vma->vm_mm, vma->vm_start);
> > /* With radix we don't use slice, so derive it from vma*/
> > - if (!radix_enabled())
> > + if (!radix_enabled()) {
> > + unsigned int psize = get_slice_psize(vma->vm_mm, vma->vm_start);
>
> Insert a blank line here.
Okay.
Thanks for the review, it's really appreciated.
Thanks,
Nick
^ permalink raw reply
* Re: [PATCH 08/10] powerpc/mm/slice: Use const pointers to cached slice masks where possible
From: Nicholas Piggin @ 2018-03-06 23:33 UTC (permalink / raw)
To: Christophe LEROY; +Cc: linuxppc-dev, Aneesh Kumar K . V
In-Reply-To: <248c0062-7710-bb96-6677-3c6d7bdc13ee@c-s.fr>
On Tue, 6 Mar 2018 15:55:04 +0100
Christophe LEROY <christophe.leroy@c-s.fr> wrote:
> Le 06/03/2018 à 14:25, Nicholas Piggin a écrit :
> > @@ -572,11 +555,19 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
> > #ifdef CONFIG_PPC_64K_PAGES
> > /* If we support combo pages, we can allow 64k pages in 4k slices */
> > if (psize == MMU_PAGE_64K) {
> > - compat_mask = *slice_mask_for_size(mm, MMU_PAGE_4K);
> > + compat_maskp = slice_mask_for_size(mm, MMU_PAGE_4K);
> > if (fixed)
> > - slice_or_mask(&good_mask, &good_mask, &compat_mask);
> > - }
> > + slice_or_mask(&good_mask, maskp, compat_maskp);
> > + else
> > + slice_copy_mask(&good_mask, maskp);
> > + } else
> > #endif
> > + {
> > + slice_copy_mask(&good_mask, maskp);
> > + }
>
> You could get something nicer by removing that #ifdef and doing instead:
>
> if (IS_ENABLED(CONFIG_PPC_64K_PAGES) && psize == MMU_PAGE_64K) {
> ...
> } else {
> slice_copy_mask(&good_mask, maskp);
> }
Yeah that's nicer.
> >
> > -#if 0 /* too verbose */
> > - slice_dbg("is_hugepage_only_range(mm=%p, addr=%lx, len=%lx)\n",
> > - mm, addr, len);
> > - slice_print_mask(" mask", &mask);
> > - slice_print_mask(" available", &available);
> > -#endif
>
> That's cleanup, should be in a previous patch.
Okay.
Thanks,
Nick
^ permalink raw reply
* Re: [PATCH 09/10] powerpc/mm/slice: use the dynamic high slice size to limit bitmap operations
From: Nicholas Piggin @ 2018-03-06 23:32 UTC (permalink / raw)
To: Christophe LEROY; +Cc: linuxppc-dev, Aneesh Kumar K . V
In-Reply-To: <525f5482-550e-4978-3367-feee257d4023@c-s.fr>
On Tue, 6 Mar 2018 16:02:20 +0100
Christophe LEROY <christophe.leroy@c-s.fr> wrote:
> Le 06/03/2018 à 14:25, Nicholas Piggin a écrit :
> > The number of high slices a process might use now depends on its
> > address space size, and what allocation address it has requested.
> >
> > This patch uses that limit throughout call chains where possible,
> > rather than use the fixed SLICE_NUM_HIGH for bitmap operations.
> > This saves some cost for processes that don't use very large address
> > spaces.
> >
> > Perormance numbers aren't changed significantly, this may change
> > with larger address spaces or different mmap access patterns that
> > require more slice mask building.
> >
> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> > ---
> > arch/powerpc/mm/slice.c | 75 +++++++++++++++++++++++++++++--------------------
> > 1 file changed, 45 insertions(+), 30 deletions(-)
> >
> > diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
> > index 086c31b8b982..507d17e2cfcd 100644
> > --- a/arch/powerpc/mm/slice.c
> > +++ b/arch/powerpc/mm/slice.c
> > @@ -61,14 +61,12 @@ static void slice_print_mask(const char *label, const struct slice_mask *mask) {
> > #endif
> >
> > static void slice_range_to_mask(unsigned long start, unsigned long len,
> > - struct slice_mask *ret)
> > + struct slice_mask *ret,
> > + unsigned long high_slices)
> > {
> > unsigned long end = start + len - 1;
> >
> > ret->low_slices = 0;
> > - if (SLICE_NUM_HIGH)
> > - bitmap_zero(ret->high_slices, SLICE_NUM_HIGH);
> > -
> > if (start < SLICE_LOW_TOP) {
> > unsigned long mend = min(end,
> > (unsigned long)(SLICE_LOW_TOP - 1));
> > @@ -77,6 +75,10 @@ static void slice_range_to_mask(unsigned long start, unsigned long len,
> > - (1u << GET_LOW_SLICE_INDEX(start));
> > }
> >
> > + if (!SLICE_NUM_HIGH)
> > + return;
> > +
> > + bitmap_zero(ret->high_slices, high_slices);
>
> In include/linux/bitmap.h, it is said:
>
> * Note that nbits should be always a compile time evaluable constant.
> * Otherwise many inlines will generate horrible code.
>
> Not sure that's true, but it is written ...
Good question, I'll check that.
> > static inline void slice_or_mask(struct slice_mask *dst,
> > const struct slice_mask *src1,
> > - const struct slice_mask *src2)
> > + const struct slice_mask *src2,
> > + unsigned long high_slices)
> > {
> > dst->low_slices = src1->low_slices | src2->low_slices;
> > if (!SLICE_NUM_HIGH)
> > return;
> > - bitmap_or(dst->high_slices, src1->high_slices, src2->high_slices, SLICE_NUM_HIGH);
> > + bitmap_or(dst->high_slices, src1->high_slices, src2->high_slices,
> > + high_slices);
>
> Why a new line here, this line is shorter than before.
> Or that was forgotten in a previous patch ?
Yeah it was previously a longer line. I will fix those.
> > @@ -643,17 +652,17 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
> > if (addr == -ENOMEM)
> > return -ENOMEM;
> >
> > - slice_range_to_mask(addr, len, &potential_mask);
> > + slice_range_to_mask(addr, len, &potential_mask, high_slices);
> > slice_dbg(" found potential area at 0x%lx\n", addr);
> > slice_print_mask(" mask", &potential_mask);
> >
> > convert:
> > - slice_andnot_mask(&potential_mask, &potential_mask, &good_mask);
> > + slice_andnot_mask(&potential_mask, &potential_mask, &good_mask, high_slices);
> > if (compat_maskp && !fixed)
> > - slice_andnot_mask(&potential_mask, &potential_mask, compat_maskp);
> > + slice_andnot_mask(&potential_mask, &potential_mask, compat_maskp, high_slices);
> > if (potential_mask.low_slices ||
> > (SLICE_NUM_HIGH &&
> > - !bitmap_empty(potential_mask.high_slices, SLICE_NUM_HIGH))) {
> > + !bitmap_empty(potential_mask.high_slices, high_slices))) {
>
> Are we sure high_slices is not nul here when SLICE_NUM_HIGH is not nul ?
On 64/s it should be for 64-bit processes, but perhaps not 32. I have
to look into that, so another good catch.
Perhaps I will leave this patch off the series for now because I didn't
measure much difference. Aneesh wants to expand address space even more,
so I might revisit after his patches go in, to see if the optimistation
becomes worthwhile.
Thanks,
Nick
^ permalink raw reply
* Re: [PATCH 07/10] powerpc/mm/slice: Switch to 3-operand slice bitops helpers
From: Nicholas Piggin @ 2018-03-06 23:19 UTC (permalink / raw)
To: Christophe LEROY; +Cc: linuxppc-dev, Aneesh Kumar K . V
In-Reply-To: <49fb018e-50b2-a37c-e329-19b5d4f87942@c-s.fr>
On Tue, 6 Mar 2018 15:44:46 +0100
Christophe LEROY <christophe.leroy@c-s.fr> wrote:
> Le 06/03/2018 à 14:25, Nicholas Piggin a écrit :
> > This converts the slice_mask bit operation helpers to be the usual
> > 3-operand kind, which is clearer to work with.
>
> What's the real benefit of doing that ?
One or two places where we want to combine 2 const input bitmaps
and can avoid the extra copy in the next patch.
E.g., slice_or_mask(&good_mask, maskp, compat_maskp);
>
> It is helps for a subsequent patch, say it.
Fair point, I'll add to the changelog.
> > @@ -433,25 +433,33 @@ static unsigned long slice_find_area(struct mm_struct *mm, unsigned long len,
> > return slice_find_area_bottomup(mm, len, mask, psize, high_limit);
> > }
> >
> > -static inline void slice_or_mask(struct slice_mask *dst,
> > +static inline void slice_copy_mask(struct slice_mask *dst,
> > const struct slice_mask *src)
>
> This new function is not used, the compiler while probably not be happy.
I think it doesn't get so grumpy with inlines (otherwise we'd have a lot
of problems of headers). Usually I think the new function should go into
the patch where it's first used, but this being a self contained helper,
I thought it fit better to add here. Maybe I'm wrong, I can move it.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox