Commit 722bc6b167 breaks booting on an 8TB system.

Commit 722bc6b167 breaks booting on an 8TB system.

[Robin Holt]

I just tried to boot an 8TB system.  It fails very early in boot with:

Kernel panic - not syncing: Cannot find space for the kernel page tables


A git bisect lead me to commit 722bc6b16771ed80871e1fd81c86d3627dda2ac8.

A git revert of that commit based on Linus' current as of a few hours ago
(commit fb21aff) does boot past that point on the 8TB configuration.

Thanks,
Robin

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Ethan.kernel@gmail]

very interested about how long time take it to boot,
I have tested with 4TB.

Thanks for your info.

Ethan

在 2012-6-1，23:54，Robin Holt <holt@sgi.com> 写道：

> I just tried to boot an 8TB system.  It fails very early in boot with:
> 
> Kernel panic - not syncing: Cannot find space for the kernel page tables
> 
> 
> A git bisect lead me to commit 722bc6b16771ed80871e1fd81c86d3627dda2ac8.
> 
> A git revert of that commit based on Linus' current as of a few hours ago
> (commit fb21aff) does boot past that point on the 8TB configuration.
> 
> Thanks,
> Robin
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Yinghai Lu]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On Fri, Jun 1, 2012 at 8:54 AM, Robin Holt <holt@sgi.com> wrote:
> I just tried to boot an 8TB system.  It fails very early in boot with:
>
> Kernel panic - not syncing: Cannot find space for the kernel page tables
>
>
> A git bisect lead me to commit 722bc6b16771ed80871e1fd81c86d3627dda2ac8.
>
> A git revert of that commit based on Linus' current as of a few hours ago
> (commit fb21aff) does boot past that point on the 8TB configuration.

have some local patches. please check if they are helping...

Thanks

Yinghai

[-- Attachment #2: only_need_recalculate_for_low.patch --]
[-- Type: application/octet-stream, Size: 553 bytes --]

---
 arch/x86/mm/init.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-2.6/arch/x86/mm/init.c
===================================================================
--- linux-2.6.orig/arch/x86/mm/init.c
+++ linux-2.6/arch/x86/mm/init.c
@@ -62,7 +62,8 @@ static void __init find_early_table_spac
 		extra += PMD_SIZE;
 #endif
 		/* The first 2/4M doesn't use large pages. */
-		extra += mr->end - mr->start;
+		if (mr->start < PMD_SIZE)
+			extra += mr->end - mr->start;
 
 		ptes = (extra + PAGE_SIZE - 1) >> PAGE_SHIFT;
 	} else

[-- Attachment #3: get_page_size_mask.patch --]
[-- Type: application/octet-stream, Size: 4696 bytes --]

Subject: [PATCH 1/3] x86, mm:  Introduce global page_size_mask

Add probe_page_size_mask() to detect if need to use 1G or 2M.
and store them in page_size_mask.

Only probe them at first init_memory_mapping calling.
second and later init_memory_mapping() calling does not need probe again.
also we don't need to pass use_gbpages around.

Suggested-by: Ingo Molnar <mingo@elte.hu>
Signe-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/include/asm/pgtable.h |    1 
 arch/x86/mm/init.c             |   70 +++++++++++++++++++++--------------------
 2 files changed, 37 insertions(+), 34 deletions(-)

Index: linux-2.6/arch/x86/include/asm/pgtable.h
===================================================================
--- linux-2.6.orig/arch/x86/include/asm/pgtable.h
+++ linux-2.6/arch/x86/include/asm/pgtable.h
@@ -597,6 +597,7 @@ static inline int pgd_none(pgd_t pgd)
 #ifndef __ASSEMBLY__
 
 extern int direct_gbpages;
+extern int page_size_mask;
 
 /* local pte updates need not use xchg for locking */
 static inline pte_t native_local_ptep_get_and_clear(pte_t *ptep)
Index: linux-2.6/arch/x86/mm/init.c
===================================================================
--- linux-2.6.orig/arch/x86/mm/init.c
+++ linux-2.6/arch/x86/mm/init.c
@@ -35,8 +35,10 @@ struct map_range {
 	unsigned page_size_mask;
 };
 
-static void __init find_early_table_space(struct map_range *mr, unsigned long end,
-					  int use_pse, int use_gbpages)
+int page_size_mask = -1;
+
+static void __init find_early_table_space(struct map_range *mr,
+					  unsigned long end)
 {
 	unsigned long puds, pmds, ptes, tables, start = 0, good_end = end;
 	phys_addr_t base;
@@ -44,7 +46,7 @@ static void __init find_early_table_spac
 	puds = (end + PUD_SIZE - 1) >> PUD_SHIFT;
 	tables = roundup(puds * sizeof(pud_t), PAGE_SIZE);
 
-	if (use_gbpages) {
+	if (page_size_mask & (1 << PG_LEVEL_1G)) {
 		unsigned long extra;
 
 		extra = end - ((end>>PUD_SHIFT) << PUD_SHIFT);
@@ -54,7 +56,7 @@ static void __init find_early_table_spac
 
 	tables += roundup(pmds * sizeof(pmd_t), PAGE_SIZE);
 
-	if (use_pse) {
+	if (page_size_mask & (1 << PG_LEVEL_2M)) {
 		unsigned long extra;
 
 		extra = end - ((end>>PMD_SHIFT) << PMD_SHIFT);
@@ -90,6 +92,34 @@ static void __init find_early_table_spac
 		(pgt_buf_top << PAGE_SHIFT) - 1);
 }
 
+static void probe_page_size_mask(void)
+{
+	if (page_size_mask != -1)
+		return;
+
+	page_size_mask = 0;
+#if !defined(CONFIG_DEBUG_PAGEALLOC) && !defined(CONFIG_KMEMCHECK)
+	/*
+	 * For CONFIG_DEBUG_PAGEALLOC, identity mapping will use small pages.
+	 * This will simplify cpa(), which otherwise needs to support splitting
+	 * large pages into small in interrupt context, etc.
+	 */
+	if (direct_gbpages)
+		page_size_mask |= 1 << PG_LEVEL_1G;
+	if (cpu_has_pse)
+		page_size_mask |= 1 << PG_LEVEL_2M;
+#endif
+
+	/* Enable PSE if available */
+	if (cpu_has_pse)
+		set_in_cr4(X86_CR4_PSE);
+
+	/* Enable PGE if available */
+	if (cpu_has_pge) {
+		set_in_cr4(X86_CR4_PGE);
+		__supported_pte_mask |= _PAGE_GLOBAL;
+	}
+}
 void __init native_pagetable_reserve(u64 start, u64 end)
 {
 	memblock_reserve(start, end - start);
@@ -125,44 +155,16 @@ static int __meminit save_mr(struct map_
 unsigned long __init_refok init_memory_mapping(unsigned long start,
 					       unsigned long end)
 {
-	unsigned long page_size_mask = 0;
 	unsigned long start_pfn, end_pfn;
 	unsigned long ret = 0;
 	unsigned long pos;
-
 	struct map_range mr[NR_RANGE_MR];
 	int nr_range, i;
-	int use_pse, use_gbpages;
 
 	printk(KERN_INFO "init_memory_mapping: [mem %#010lx-%#010lx]\n",
 	       start, end - 1);
 
-#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_KMEMCHECK)
-	/*
-	 * For CONFIG_DEBUG_PAGEALLOC, identity mapping will use small pages.
-	 * This will simplify cpa(), which otherwise needs to support splitting
-	 * large pages into small in interrupt context, etc.
-	 */
-	use_pse = use_gbpages = 0;
-#else
-	use_pse = cpu_has_pse;
-	use_gbpages = direct_gbpages;
-#endif
-
-	/* Enable PSE if available */
-	if (cpu_has_pse)
-		set_in_cr4(X86_CR4_PSE);
-
-	/* Enable PGE if available */
-	if (cpu_has_pge) {
-		set_in_cr4(X86_CR4_PGE);
-		__supported_pte_mask |= _PAGE_GLOBAL;
-	}
-
-	if (use_gbpages)
-		page_size_mask |= 1 << PG_LEVEL_1G;
-	if (use_pse)
-		page_size_mask |= 1 << PG_LEVEL_2M;
+	probe_page_size_mask();
 
 	memset(mr, 0, sizeof(mr));
 	nr_range = 0;
@@ -267,7 +269,7 @@ unsigned long __init_refok init_memory_m
 	 * nodes are discovered.
 	 */
 	if (!after_bootmem)
-		find_early_table_space(&mr[0], end, use_pse, use_gbpages);
+		find_early_table_space(&mr[0], end);
 
 	for (i = 0; i < nr_range; i++)
 		ret = kernel_physical_mapping_init(mr[i].start, mr[i].end,

[-- Attachment #4: mr_cal.patch --]
[-- Type: application/octet-stream, Size: 2139 bytes --]

---
 arch/x86/mm/init.c |   46 ++++++++++++++++++++++++++++------------------
 1 file changed, 28 insertions(+), 18 deletions(-)

Index: linux-2.6/arch/x86/mm/init.c
===================================================================
--- linux-2.6.orig/arch/x86/mm/init.c
+++ linux-2.6/arch/x86/mm/init.c
@@ -147,27 +147,13 @@ static int __meminit save_mr(struct map_
 	return nr_range;
 }
 
-/*
- * Setup the direct mapping of the physical memory at PAGE_OFFSET.
- * This runs before bootmem is initialized and gets pages directly from
- * the physical memory. To access them they are temporarily mapped.
- */
-unsigned long __init_refok init_memory_mapping(unsigned long start,
-					       unsigned long end)
+static int __meminit split_mem_range(struct map_range *mr, int nr_range,
+				     unsigned long start,
+				     unsigned long end)
 {
 	unsigned long start_pfn, end_pfn;
-	unsigned long ret = 0;
 	unsigned long pos;
-	struct map_range mr[NR_RANGE_MR];
-	int nr_range, i;
-
-	printk(KERN_INFO "init_memory_mapping: [mem %#010lx-%#010lx]\n",
-	       start, end - 1);
-
-	probe_page_size_mask();
-
-	memset(mr, 0, sizeof(mr));
-	nr_range = 0;
+	int i;
 
 	/* head if not big page alignment ? */
 	start_pfn = start >> PAGE_SHIFT;
@@ -261,6 +247,30 @@ unsigned long __init_refok init_memory_m
 			(mr[i].page_size_mask & (1<<PG_LEVEL_1G))?"1G":(
 			 (mr[i].page_size_mask & (1<<PG_LEVEL_2M))?"2M":"4k"));
 
+	return nr_range;
+}
+
+/*
+ * Setup the direct mapping of the physical memory at PAGE_OFFSET.
+ * This runs before bootmem is initialized and gets pages directly from
+ * the physical memory. To access them they are temporarily mapped.
+ */
+unsigned long __init_refok init_memory_mapping(unsigned long start,
+					       unsigned long end)
+{
+	struct map_range mr[NR_RANGE_MR];
+	unsigned long ret = 0;
+	int nr_range, i;
+
+	printk(KERN_INFO "init_memory_mapping: [mem %#010lx-%#010lx]\n",
+	       start, end - 1);
+
+	probe_page_size_mask();
+
+	memset(mr, 0, sizeof(mr));
+	nr_range = 0;
+	nr_range = split_mem_range(mr, nr_range, start, end);
+
 	/*
 	 * Find space for the kernel direct mapping tables.
 	 *

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Robin Holt]

On Sun, Jun 03, 2012 at 12:00:29AM -0700, Yinghai Lu wrote:
> On Fri, Jun 1, 2012 at 8:54 AM, Robin Holt <holt@sgi.com> wrote:
> > I just tried to boot an 8TB system.  It fails very early in boot with:
> >
> > Kernel panic - not syncing: Cannot find space for the kernel page tables
> >
> >
> > A git bisect lead me to commit 722bc6b16771ed80871e1fd81c86d3627dda2ac8.
> >
> > A git revert of that commit based on Linus' current as of a few hours ago
> > (commit fb21aff) does boot past that point on the 8TB configuration.
> 
> have some local patches. please check if they are helping...

I will not have access to the machine until late tonight/early tomorrow
(approximately 24 hours from now).  I will try to give the patches a
run then.  Would you like them tested individually or as a set?

Robin

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Yinghai Lu]

On Mon, Jun 4, 2012 at 3:07 AM, Robin Holt <holt@sgi.com> wrote:
> On Sun, Jun 03, 2012 at 12:00:29AM -0700, Yinghai Lu wrote:
>> On Fri, Jun 1, 2012 at 8:54 AM, Robin Holt <holt@sgi.com> wrote:
>> > I just tried to boot an 8TB system.  It fails very early in boot with:
>> >
>> > Kernel panic - not syncing: Cannot find space for the kernel page tables
>> >
>> >
>> > A git bisect lead me to commit 722bc6b16771ed80871e1fd81c86d3627dda2ac8.
>> >
>> > A git revert of that commit based on Linus' current as of a few hours ago
>> > (commit fb21aff) does boot past that point on the 8TB configuration.
>>
>> have some local patches. please check if they are helping...
>
> I will not have access to the machine until late tonight/early tomorrow
> (approximately 24 hours from now).  I will try to give the patches a
> run then.  Would you like them tested individually or as a set?

only_need_recalculate_for_low.patch

should be enough.

Thanks

Yinghai

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Robin Holt]

On Mon, Jun 04, 2012 at 04:51:19PM -0700, Yinghai Lu wrote:
> On Mon, Jun 4, 2012 at 3:07 AM, Robin Holt <holt@sgi.com> wrote:
> > On Sun, Jun 03, 2012 at 12:00:29AM -0700, Yinghai Lu wrote:
> >> On Fri, Jun 1, 2012 at 8:54 AM, Robin Holt <holt@sgi.com> wrote:
> >> > I just tried to boot an 8TB system.  It fails very early in boot with:
> >> >
> >> > Kernel panic - not syncing: Cannot find space for the kernel page tables
> >> >
> >> >
> >> > A git bisect lead me to commit 722bc6b16771ed80871e1fd81c86d3627dda2ac8.
> >> >
> >> > A git revert of that commit based on Linus' current as of a few hours ago
> >> > (commit fb21aff) does boot past that point on the 8TB configuration.
> >>
> >> have some local patches. please check if they are helping...
> >
> > I will not have access to the machine until late tonight/early tomorrow
> > (approximately 24 hours from now).  I will try to give the patches a
> > run then.  Would you like them tested individually or as a set?
> 
> only_need_recalculate_for_low.patch
> 
> should be enough.

With just that patch, no improvement.

Sorry about the delay.  I got bumped from the machine yesterday.
The next reservable time is tomorrow night, otherwise, the machine is
booked solid for the next 36 hours.

Robin

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Robin Holt]

> With just that patch, no improvement.
> 
> Sorry about the delay.  I got bumped from the machine yesterday.
> The next reservable time is tomorrow night, otherwise, the machine is
> booked solid for the next 36 hours.

Wait.  I was wrong.  I booted the wrong kernel.  With that patch applied,
the machine does boot.

Thanks,
Robin

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Ingo Molnar]

* Robin Holt <holt@sgi.com> wrote:

> > With just that patch, no improvement.
> > 
> > Sorry about the delay.  I got bumped from the machine yesterday.
> > The next reservable time is tomorrow night, otherwise, the machine is
> > booked solid for the next 36 hours.
> 
> Wait.  I was wrong.  I booted the wrong kernel.  With that 
> patch applied, the machine does boot.

Great.

Could you guys please write a usable changelog for 
only_need_recalculate_for_low.patch and send it to me with lkml 
Cc:-ed, so that I can send the fix to Linus ASAP?

Thanks,

	Ingo

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Robin Holt]

On Wed, Jun 06, 2012 at 11:51:02AM +0200, Ingo Molnar wrote:
> 
> * Robin Holt <holt@sgi.com> wrote:
> 
> > > With just that patch, no improvement.
> > > 
> > > Sorry about the delay.  I got bumped from the machine yesterday.
> > > The next reservable time is tomorrow night, otherwise, the machine is
> > > booked solid for the next 36 hours.
> > 
> > Wait.  I was wrong.  I booted the wrong kernel.  With that 
> > patch applied, the machine does boot.
> 
> Great.
> 
> Could you guys please write a usable changelog for 
> only_need_recalculate_for_low.patch and send it to me with lkml 
> Cc:-ed, so that I can send the fix to Linus ASAP?

I also tested this on a separate 2TB system which had been experiencing
boot problems as well as a the original failing system configured as a
16TB system.  It worked on all systems tested so far.

Robin

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Yinghai Lu]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

On Wed, Jun 6, 2012 at 2:51 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Robin Holt <holt@sgi.com> wrote:
>>
>> Wait.  I was wrong.  I booted the wrong kernel.  With that
>> patch applied, the machine does boot.
>
> Great.
>
> Could you guys please write a usable changelog for
> only_need_recalculate_for_low.patch and send it to me with lkml
> Cc:-ed, so that I can send the fix to Linus ASAP?

please check attached one.

Thanks

Yinghai

[-- Attachment #2: only_need_recalculate_for_low_v2.patch --]
[-- Type: application/octet-stream, Size: 1235 bytes --]

Subject: [PATCH] x86: Only add extra pages count for first memory range during pre-allocate early page table space

Robin found:
| I just tried to boot an 8TB system.  It fails very early in boot with:
|
| Kernel panic - not syncing: Cannot find space for the kernel page tables

git bisect commit 722bc6b16771ed80871e1fd81c86d3627dda2ac8.
A git revert of that commit does boot past that point on the 8TB configuration.

That commit will add up extra pages for all memory range even above 4g.

Try to limit that extra page count adding to first entry only.

Bisected-by: Robin Holt <holt@sgi.com>
Tested-by: Robin Holt <holt@sgi.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/mm/init.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-2.6/arch/x86/mm/init.c
===================================================================
--- linux-2.6.orig/arch/x86/mm/init.c
+++ linux-2.6/arch/x86/mm/init.c
@@ -62,7 +62,8 @@ static void __init find_early_table_spac
 		extra += PMD_SIZE;
 #endif
 		/* The first 2/4M doesn't use large pages. */
-		extra += mr->end - mr->start;
+		if (mr->start < PMD_SIZE)
+			extra += mr->end - mr->start;
 
 		ptes = (extra + PAGE_SIZE - 1) >> PAGE_SHIFT;
 	} else

Re: Commit 722bc6b167 breaks booting on an 8TB system.

[Robin Holt]

On Wed, Jun 06, 2012 at 10:55:40AM -0700, Yinghai Lu wrote:
> On Wed, Jun 6, 2012 at 2:51 AM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > * Robin Holt <holt@sgi.com> wrote:
> >>
> >> Wait.  I was wrong.  I booted the wrong kernel.  With that
> >> patch applied, the machine does boot.
> >
> > Great.
> >
> > Could you guys please write a usable changelog for
> > only_need_recalculate_for_low.patch and send it to me with lkml
> > Cc:-ed, so that I can send the fix to Linus ASAP?
> 
> please check attached one.


Your attached patch is the same as the patch I had tested.  I do not
have access to the machine for more than a day to test this actual patch,
but as far as 'diff' can tell, the difference is commit header only.

Robin

[tip:x86/urgent] x86/mm: Only add extra pages count for the first memory range during pre-allocation early page table space

[tip-bot for Yinghai Lu]

Commit-ID:  bd2753b2dda7bb43c7468826de75f49c6a7e8965
Gitweb:     http://git.kernel.org/tip/bd2753b2dda7bb43c7468826de75f49c6a7e8965
Author:     Yinghai Lu <yinghai@kernel.org>
AuthorDate: Wed, 6 Jun 2012 10:55:40 -0700
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Fri, 8 Jun 2012 11:40:50 +0200

x86/mm: Only add extra pages count for the first memory range during pre-allocation early page table space

Robin found this regression:

| I just tried to boot an 8TB system.  It fails very early in boot with:
| Kernel panic - not syncing: Cannot find space for the kernel page tables

git bisect commit 722bc6b16771ed80871e1fd81c86d3627dda2ac8.

A git revert of that commit does boot past that point on the 8TB
configuration.

That commit will add up extra pages for all memory range even
above 4g.

Try to limit that extra page count adding to first entry only.

Bisected-by: Robin Holt <holt@sgi.com>
Tested-by: Robin Holt <holt@sgi.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Cc: WANG Cong <xiyou.wangcong@gmail.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/CAE9FiQUj3wyzQxtq9yzBNc9u220p8JZ1FYHG7t%3DMOzJ%3D9BZMYA@mail.gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/mm/init.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
index 97141c2..bc4e9d8 100644
--- a/arch/x86/mm/init.c
+++ b/arch/x86/mm/init.c
@@ -62,7 +62,8 @@ static void __init find_early_table_space(struct map_range *mr, unsigned long en
 		extra += PMD_SIZE;
 #endif
 		/* The first 2/4M doesn't use large pages. */
-		extra += mr->end - mr->start;
+		if (mr->start < PMD_SIZE)
+			extra += mr->end - mr->start;
 
 		ptes = (extra + PAGE_SIZE - 1) >> PAGE_SHIFT;
 	} else