[PATCHv3 0/7] arm64: Use contiguous PTE bit for the kernel linear mapping

[PATCHv3 0/7] arm64: Use contiguous PTE bit for the kernel linear mapping

[Jeremy Linton]

Changes from V2:

This version removes the kernel command line parameter and replaces it with a
config option. It also renames a couple of the macros. Patches rebased to 4.3rc4.

Changes from V1:

This version is similar to the previous versions with the addition of
contpte/nocontpte kernel parameters for enabling/disabling the functionality.
I defaulted it to contpte, but will change it to nocontpte if someone feels
strongly about that. I considered making the default a kconfig option, but
decided against it to avoid adding clutter for what should eventually be a
fairly normal piece of functionality. I also used the early_param_on_off()
macro, which is a small and concise piece of code, but varies a little from
the more common parameter=value kernel parameters. 

Also, continuing thanks to Steve Capper who has been religiously testing these patch sets,
and found a bug in the previous implementation.

Finally, I removed the formatting change per Catalin.

Original Description:

When running 64k pages it became apparent that it was possible that the
kernel could create more TLB pressure in certain circumstances when
compared to the 4k page kernel. This is due to the fact that large
portions of the 4k kernel can be mapped with 2M blocks.
While the larger block size for 64k pages is 512M, it 
frequently won't be used for assorted reasons.

ARMv8 has a contiguous PTE bit that allows the TLBs to map a range larger
than a single PTE if the range is physically contiguous. So its
a good idea to use this where appropriate for the kernel mapping.

This patch adds the definitions for the contiguous bit, updates the kernel
page range dump to understand them (and block translations),
and updates the code in mmu.c to use the contiguous bit where appropriate
for the kernel linear mapping.

Jeremy Linton (7):
  arm64: Add contiguous page flag shifts and constants
  arm64: PTE/PMD contiguous bit definition
  arm64: Macros to check/set/unset the contiguous bit
  arm64: Default kernel pages should be contiguous
  arm64: Make the kernel page dump utility aware of the CONT bit
  arm64: Add a kernel Kconfig option to enable contiguous PTE marking
  arm64: Mark kernel page ranges contiguous

 arch/arm64/Kconfig                     |  7 +++
 arch/arm64/include/asm/page.h          |  8 +++-
 arch/arm64/include/asm/pgtable-hwdef.h |  9 ++++
 arch/arm64/include/asm/pgtable.h       | 13 ++++++
 arch/arm64/mm/dump.c                   | 18 +++++++-
 arch/arm64/mm/mmu.c                    | 81 ++++++++++++++++++++++++++++++----
 6 files changed, 125 insertions(+), 11 deletions(-)

-- 
2.4.3

[PATCH 1/7] arm64: Add contiguous page flag shifts and constants

[Jeremy Linton]

Add the number of pages required to form a contiguous range,
as well as some supporting constants.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/include/asm/page.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index 7d9c7e4..33892ef 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -20,14 +20,20 @@
 #define __ASM_PAGE_H
 
 /* PAGE_SHIFT determines the page size */
+/* CONT_SHIFT determines the number of pages which can be tracked together  */
 #ifdef CONFIG_ARM64_64K_PAGES
 #define PAGE_SHIFT		16
+#define CONT_SHIFT		5
 #else
 #define PAGE_SHIFT		12
+#define CONT_SHIFT		4
 #endif
-#define PAGE_SIZE		(_AC(1,UL) << PAGE_SHIFT)
+#define PAGE_SIZE		(_AC(1, UL) << PAGE_SHIFT)
 #define PAGE_MASK		(~(PAGE_SIZE-1))
 
+#define CONT_SIZE		(_AC(1, UL) << (CONT_SHIFT + PAGE_SHIFT))
+#define CONT_MASK		(~(CONT_SIZE-1))
+
 /*
  * The idmap and swapper page tables need some space reserved in the kernel
  * image. Both require pgd, pud (4 levels only) and pmd tables to (section)
-- 
2.4.3

[PATCH 2/7] arm64: PTE/PMD contiguous bit definition

[Jeremy Linton]

Define the bit positions in the PTE and PMD for the
contiguous bit.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/include/asm/pgtable-hwdef.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index 24154b0..95c1ec0 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -55,6 +55,13 @@
 #define SECTION_MASK		(~(SECTION_SIZE-1))
 
 /*
+ * Contiguous page definitions.
+ */
+#define CONT_PTES		(_AC(1, UL) << CONT_SHIFT)
+/* the the numerical offset of the PTE within a range of CONT_PTES */
+#define CONT_RANGE_OFFSET(addr) (((addr)>>PAGE_SHIFT)&(CONT_PTES-1))
+
+/*
  * Hardware page table definitions.
  *
  * Level 1 descriptor (PUD).
@@ -83,6 +90,7 @@
 #define PMD_SECT_S		(_AT(pmdval_t, 3) << 8)
 #define PMD_SECT_AF		(_AT(pmdval_t, 1) << 10)
 #define PMD_SECT_NG		(_AT(pmdval_t, 1) << 11)
+#define PMD_SECT_CONT		(_AT(pmdval_t, 1) << 52)
 #define PMD_SECT_PXN		(_AT(pmdval_t, 1) << 53)
 #define PMD_SECT_UXN		(_AT(pmdval_t, 1) << 54)
 
@@ -105,6 +113,7 @@
 #define PTE_AF			(_AT(pteval_t, 1) << 10)	/* Access Flag */
 #define PTE_NG			(_AT(pteval_t, 1) << 11)	/* nG */
 #define PTE_DBM			(_AT(pteval_t, 1) << 51)	/* Dirty Bit Management */
+#define PTE_CONT		(_AT(pteval_t, 1) << 52)	/* Contiguous range */
 #define PTE_PXN			(_AT(pteval_t, 1) << 53)	/* Privileged XN */
 #define PTE_UXN			(_AT(pteval_t, 1) << 54)	/* User XN */
 
-- 
2.4.3

[PATCH 3/7] arm64: Macros to check/set/unset the contiguous bit

[Jeremy Linton]

Add the supporting macros to check if the contiguous bit
is set, set the bit, or clear it in a PTE entry.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/include/asm/pgtable.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 26b0666..cc19ec1 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -140,6 +140,7 @@ extern struct page *empty_zero_page;
 #define pte_special(pte)	(!!(pte_val(pte) & PTE_SPECIAL))
 #define pte_write(pte)		(!!(pte_val(pte) & PTE_WRITE))
 #define pte_exec(pte)		(!(pte_val(pte) & PTE_UXN))
+#define pte_cont(pte)		(!!(pte_val(pte) & PTE_CONT))
 
 #ifdef CONFIG_ARM64_HW_AFDBM
 #define pte_hw_dirty(pte)	(pte_write(pte) && !(pte_val(pte) & PTE_RDONLY))
@@ -202,6 +203,16 @@ static inline pte_t pte_mkspecial(pte_t pte)
 	return set_pte_bit(pte, __pgprot(PTE_SPECIAL));
 }
 
+static inline pte_t pte_mkcont(pte_t pte)
+{
+	return set_pte_bit(pte, __pgprot(PTE_CONT));
+}
+
+static inline pte_t pte_mknoncont(pte_t pte)
+{
+	return clear_pte_bit(pte, __pgprot(PTE_CONT));
+}
+
 static inline void set_pte(pte_t *ptep, pte_t pte)
 {
 	*ptep = pte;
-- 
2.4.3

[PATCH 4/7] arm64: Default kernel pages should be contiguous

[Jeremy Linton]

The default page attributes for a PMD being broken should have the CONT bit
set. Create a new definition for an early boot range of PTE's that are
contiguous.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/include/asm/pgtable.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index cc19ec1..a1bcd64 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -72,6 +72,8 @@ extern void __pgd_error(const char *file, int line, unsigned long val);
 
 #define PAGE_KERNEL		__pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE)
 #define PAGE_KERNEL_EXEC	__pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE)
+#define PAGE_KERNEL_EXEC_CONT	__pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | \
+					 PTE_WRITE | PTE_CONT)
 
 #define PAGE_HYP		__pgprot(_PAGE_DEFAULT | PTE_HYP)
 #define PAGE_HYP_DEVICE		__pgprot(PROT_DEVICE_nGnRE | PTE_HYP)
-- 
2.4.3

[PATCH 5/7] arm64: Make the kernel page dump utility aware of the CONT bit

[Jeremy Linton]

The kernel page dump utility needs to be aware of the CONT bit before
it will break up pages ranges for display.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/mm/dump.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/mm/dump.c b/arch/arm64/mm/dump.c
index f3d6221..5a22a11 100644
--- a/arch/arm64/mm/dump.c
+++ b/arch/arm64/mm/dump.c
@@ -67,6 +67,12 @@ static struct addr_marker address_markers[] = {
 	{ -1,			NULL },
 };
 
+/*
+ * The page dumper groups page table entries of the same type into a single
+ * description. It uses pg_state to track the range information while
+ * iterating over the pte entries. When the continuity is broken it then
+ * dumps out a description of the range.
+ */
 struct pg_state {
 	struct seq_file *seq;
 	const struct addr_marker *marker;
@@ -114,6 +120,16 @@ static const struct prot_bits pte_bits[] = {
 		.set	= "NG",
 		.clear	= "  ",
 	}, {
+		.mask	= PTE_CONT,
+		.val	= PTE_CONT,
+		.set	= "CON",
+		.clear	= "   ",
+	}, {
+		.mask	= PTE_TABLE_BIT,
+		.val	= PTE_TABLE_BIT,
+		.set	= "   ",
+		.clear	= "BLK",
+	}, {
 		.mask	= PTE_UXN,
 		.val	= PTE_UXN,
 		.set	= "UXN",
@@ -198,7 +214,7 @@ static void note_page(struct pg_state *st, unsigned long addr, unsigned level,
 		unsigned long delta;
 
 		if (st->current_prot) {
-			seq_printf(st->seq, "0x%16lx-0x%16lx   ",
+			seq_printf(st->seq, "0x%016lx-0x%016lx   ",
 				   st->start_address, addr);
 
 			delta = (addr - st->start_address) >> 10;
-- 
2.4.3

[PATCH 6/7] arm64: Add a kernel Kconfig option to enable contiguous PTE marking

[Jeremy Linton]

Add ARM64_CONT_PTE, so that the contiguous PTE marking
in the kernel's linear mapping may be turned off.
In general this should not be necessary, but it can
help sanity check that the code/hardware is working properly.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/Kconfig | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 07d1811..9485c14 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -400,6 +400,13 @@ config ARM64_VA_BITS
 	default 42 if ARM64_VA_BITS_42
 	default 48 if ARM64_VA_BITS_48
 
+config ARM64_CONT_PTE
+	bool "Use contiguous page hint for kernel linear mapping"
+	default y
+	help
+          Marking kernel pages contiguous can reduce TLB pressure for
+	  certain workloads. If unsure, say Y.
+
 config CPU_BIG_ENDIAN
        bool "Build big-endian kernel"
        help
-- 
2.4.3

[PATCH 7/7] arm64: Mark kernel page ranges contiguous

[Jeremy Linton]

With 64k pages, the next larger segment size is 512M. The linux
kernel also uses different protection flags to cover its code and data.
Because of this requirement, the vast majority of the kernel code and
data structures end up being mapped with 64k pages instead of the larger
pages common with a 4k page kernel.

Recent ARM processors support a contiguous bit in the
page tables which allows the a TLB to cover a range larger than a
single PTE if that range is mapped into physically contiguous
ram.

So, for the kernel its a good idea to set this flag. Some basic
micro benchmarks show it can significantly reduce the number of
L1 dTLB refills.

Add boot option to enable/disable CONT marking, as well as fix a
bug found by Steve Capper.

Signed-off-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/mm/mmu.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 72 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 9211b85..031ea7a 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -69,6 +69,10 @@ static void __init *early_alloc(unsigned long sz)
 	return ptr;
 }
 
+#ifndef CONFIG_ARM64_CONT_PTE
+#define CONFIG_ARM64_CONT_PTE 0
+#endif
+
 /*
  * remap a PMD into pages
  */
@@ -80,19 +84,58 @@ static void split_pmd(pmd_t *pmd, pte_t *pte)
 	do {
 		/*
 		 * Need to have the least restrictive permissions available
-		 * permissions will be fixed up later
+		 * permissions will be fixed up later. Default the new page
+		 * range as contiguous ptes.
 		 */
-		set_pte(pte, pfn_pte(pfn, PAGE_KERNEL_EXEC));
+		if (CONFIG_ARM64_CONT_PTE)
+			set_pte(pte, pfn_pte(pfn, PAGE_KERNEL_EXEC_CONT));
+		else
+			set_pte(pte, pfn_pte(pfn, PAGE_KERNEL_EXEC));
 		pfn++;
 	} while (pte++, i++, i < PTRS_PER_PTE);
 }
 
+/*
+ * Given a PTE with the CONT bit set, determine where the CONT range
+ * starts, and clear the entire range of PTE CONT bits.
+ */
+static void clear_cont_pte_range(pte_t *pte, unsigned long addr)
+{
+	int i;
+
+	pte -= CONT_RANGE_OFFSET(addr);
+	for (i = 0; i < CONT_PTES; i++) {
+		set_pte(pte, pte_mknoncont(*pte));
+		pte++;
+	}
+	flush_tlb_all();
+}
+
+/*
+ * Given a range of PTEs set the pfn and provided page protection flags
+ */
+static void __populate_init_pte(pte_t *pte, unsigned long addr,
+				unsigned long end, phys_addr_t phys,
+				pgprot_t prot)
+{
+	unsigned long pfn = __phys_to_pfn(phys);
+
+	do {
+		/* clear all the bits except the pfn, then apply the prot */
+		set_pte(pte, pfn_pte(pfn, prot));
+		pte++;
+		pfn++;
+		addr += PAGE_SIZE;
+	} while (addr != end);
+}
+
 static void alloc_init_pte(pmd_t *pmd, unsigned long addr,
-				  unsigned long end, unsigned long pfn,
+				  unsigned long end, phys_addr_t phys,
 				  pgprot_t prot,
 				  void *(*alloc)(unsigned long size))
 {
 	pte_t *pte;
+	unsigned long next;
 
 	if (pmd_none(*pmd) || pmd_sect(*pmd)) {
 		pte = alloc(PTRS_PER_PTE * sizeof(pte_t));
@@ -104,10 +147,31 @@ static void alloc_init_pte(pmd_t *pmd, unsigned long addr,
 	BUG_ON(pmd_bad(*pmd));
 
 	pte = pte_offset_kernel(pmd, addr);
-	do {
-		set_pte(pte, pfn_pte(pfn, prot));
-		pfn++;
-	} while (pte++, addr += PAGE_SIZE, addr != end);
+	if (!CONFIG_ARM64_CONT_PTE)
+		__populate_init_pte(pte, addr, end, phys, prot);
+	else
+	    do {
+		next = min(end, (addr + CONT_SIZE) & CONT_MASK);
+		if (((addr | next | phys) & ~CONT_MASK) == 0) {
+			/* a block of CONT_PTES  */
+			__populate_init_pte(pte, addr, next, phys,
+					    prot | __pgprot(PTE_CONT));
+		} else {
+			/*
+			 * If the range being split is already inside of a
+			 * contiguous range but this PTE isn't going to be
+			 * contiguous, then we want to unmark the adjacent
+			 * ranges, then update the portion of the range we
+			 * are interrested in.
+			 */
+			 clear_cont_pte_range(pte, addr);
+			 __populate_init_pte(pte, addr, next, phys, prot);
+		}
+
+		pte += (next - addr) >> PAGE_SHIFT;
+		phys += next - addr;
+		addr = next;
+	} while (addr != end);
 }
 
 void split_pud(pud_t *old_pud, pmd_t *pmd)
@@ -168,8 +232,7 @@ static void alloc_init_pmd(struct mm_struct *mm, pud_t *pud,
 				}
 			}
 		} else {
-			alloc_init_pte(pmd, addr, next, __phys_to_pfn(phys),
-				       prot, alloc);
+			alloc_init_pte(pmd, addr, next, phys, prot, alloc);
 		}
 		phys += next - addr;
 	} while (pmd++, addr = next, addr != end);
-- 
2.4.3

[PATCHv3 0/7] arm64: Use contiguous PTE bit for the kernel linear mapping

[Catalin Marinas]

On Wed, Oct 07, 2015 at 12:00:18PM -0500, Jeremy Linton wrote:
> Jeremy Linton (7):
>   arm64: Add contiguous page flag shifts and constants
>   arm64: PTE/PMD contiguous bit definition
>   arm64: Macros to check/set/unset the contiguous bit
>   arm64: Default kernel pages should be contiguous
>   arm64: Make the kernel page dump utility aware of the CONT bit
>   arm64: Add a kernel Kconfig option to enable contiguous PTE marking
>   arm64: Mark kernel page ranges contiguous

I queued the series without patch 6 (and a modification of patch 7 to
get rid of the config option check). If we need this option, we can add
it later but for (hardware) debugging purposes it looks to me like we
can simply set PTE_CONT to 0.

Thanks.

-- 
Catalin

[PATCHv3 0/7] arm64: Use contiguous PTE bit for the kernel linear mapping

[Jeremy Linton]

On 10/09/2015 07:50 AM, Catalin Marinas wrote:

> I queued the series without patch 6 (and a modification of patch 7 to
> get rid of the config option check). If we need this option, we can add
> it later but for (hardware) debugging purposes it looks to me like we
> can simply set PTE_CONT to 0.

Sounds good to me.

Thanks,

[PATCHv3 0/7] arm64: Use contiguous PTE bit for the kernel linear mapping

[Jeremy Linton]

Hello,

	Just a heads up, but this patch is causing a boot problem on a m400. I 
should know more soon.

	Thanks,

[PATCHv3 0/7] arm64: Use contiguous PTE bit for the kernel linear mapping

[Will Deacon]

On Tue, Nov 17, 2015 at 09:37:24AM -0600, Jeremy Linton wrote:
> Hello,

Hi Jeremy,

> 	Just a heads up, but this patch is causing a boot problem on a m400. I
> should know more soon.

Any chance you could share a reproducer/panic log/debug info, please?
If we can't fix this soon, then we'll have to look at reverting the change,
so getting a few more eyes on the problem might help.

Will

[PATCHv3 0/7] arm64: Use contiguous PTE bit for the kernel linear mapping

[Jeremy Linton]

On 11/18/2015 05:01 AM, Will Deacon wrote:
> On Tue, Nov 17, 2015 at 09:37:24AM -0600, Jeremy Linton wrote:
>> Hello,
>
> Hi Jeremy,
>
>> 	Just a heads up, but this patch is causing a boot problem on a m400. I
>> should know more soon.
>
> Any chance you could share a reproducer/panic log/debug info, please?
> If we can't fix this soon, then we'll have to look at reverting the change,
> so getting a few more eyes on the problem might help.

	Its usually just hangs although, once in a while it will generate an 
unknown exception. I posted a patch last night which fixes it with a big 
hammer (extra tlb flush), but It doesn't appear to have made the mailing 
list (I regularly get relay rejected messages). I will attempt to 
re-post it again.