early_node_mem()'s memory allocation policy

early_node_mem()'s memory allocation policy

[Jeremy Fitzhardinge]

 We're seeing problems under Xen where large portions of the memory
could be reserved (because they're not yet physically present, even
though the appear in E820), and the 'start' and 'end' early_node_mem()
is choosing is entirely within that reserved range.

Also, the code seems dubious because it adjusts start and end without
regarding how much space it is trying to allocate:

	/* extend the search scope */
	end = max_pfn_mapped << PAGE_SHIFT;
	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
		start = MAX_DMA32_PFN<<PAGE_SHIFT;
	else
		start = MAX_DMA_PFN<<PAGE_SHIFT;

what if max_pfn_mapped is only a few pages larger than MAX_DMA32_PFN,
and that is smaller than the size it is trying to allocate?

I tried just removing the start and end adjustments in early_node_mem()
and the kernel booted fine under Xen, but it seemed to allocate at a
very low address.  Should the for_each_active_range_index_in_nid() loop
in find_memory_core_early() be iterating from high to low addresses?  If
the allocation could be relied on to be top-down, then you wouldn't need
to adjust start at all, and it would return the highest available memory
in a natural way.

Thanks,
    J

Re: early_node_mem()'s memory allocation policy

[Yinghai Lu]

On 10/26/2010 03:18 PM, Jeremy Fitzhardinge wrote:
>  We're seeing problems under Xen where large portions of the memory
> could be reserved (because they're not yet physically present, even
> though the appear in E820), and the 'start' and 'end' early_node_mem()
> is choosing is entirely within that reserved range.
> 
> Also, the code seems dubious because it adjusts start and end without
> regarding how much space it is trying to allocate:
> 
> 	/* extend the search scope */
> 	end = max_pfn_mapped << PAGE_SHIFT;
> 	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
> 		start = MAX_DMA32_PFN<<PAGE_SHIFT;
> 	else
> 		start = MAX_DMA_PFN<<PAGE_SHIFT;
> 
> what if max_pfn_mapped is only a few pages larger than MAX_DMA32_PFN,
> and that is smaller than the size it is trying to allocate?
> 
> I tried just removing the start and end adjustments in early_node_mem()
> and the kernel booted fine under Xen, but it seemed to allocate at a
> very low address.  Should the for_each_active_range_index_in_nid() loop
> in find_memory_core_early() be iterating from high to low addresses?  If
> the allocation could be relied on to be top-down, then you wouldn't need
> to adjust start at all, and it would return the highest available memory
> in a natural way.

please check

[PATCH] x86, memblock: Fix early_node_mem with big reserved region.

Jeremy said Xen could reserve huge mem but still show as ram in e820.

early_node_mem could not find range because of start/end adjusting.

Let's use memblock_find_in_range instead ***_node. So get real top down in fallback path.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

diff --git a/arch/x86/mm/numa_64.c b/arch/x86/mm/numa_64.c
index 60f4985..7ffc9b7 100644
--- a/arch/x86/mm/numa_64.c
+++ b/arch/x86/mm/numa_64.c
@@ -178,11 +178,8 @@ static void * __init early_node_mem(int nodeid, unsigned long start,
 
 	/* extend the search scope */
 	end = max_pfn_mapped << PAGE_SHIFT;
-	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
-		start = MAX_DMA32_PFN<<PAGE_SHIFT;
-	else
-		start = MAX_DMA_PFN<<PAGE_SHIFT;
-	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
+	start = MAX_DMA_PFN << PAGE_SHIFT;
+	mem = memblock_find_in_range(start, end, size, align);
 	if (mem != MEMBLOCK_ERROR)
 		return __va(mem);
 

Re: early_node_mem()'s memory allocation policy

[Konrad Rzeszutek Wilk]

On Tue, Oct 26, 2010 at 10:49:33PM -0700, Yinghai Lu wrote:
> On 10/26/2010 03:18 PM, Jeremy Fitzhardinge wrote:
> >  We're seeing problems under Xen where large portions of the memory
> > could be reserved (because they're not yet physically present, even
> > though the appear in E820), and the 'start' and 'end' early_node_mem()
> > is choosing is entirely within that reserved range.
> > 
> > Also, the code seems dubious because it adjusts start and end without
> > regarding how much space it is trying to allocate:
> > 
> > 	/* extend the search scope */
> > 	end = max_pfn_mapped << PAGE_SHIFT;
> > 	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
> > 		start = MAX_DMA32_PFN<<PAGE_SHIFT;
> > 	else
> > 		start = MAX_DMA_PFN<<PAGE_SHIFT;
> > 
> > what if max_pfn_mapped is only a few pages larger than MAX_DMA32_PFN,
> > and that is smaller than the size it is trying to allocate?
> > 
> > I tried just removing the start and end adjustments in early_node_mem()
> > and the kernel booted fine under Xen, but it seemed to allocate at a
> > very low address.  Should the for_each_active_range_index_in_nid() loop
> > in find_memory_core_early() be iterating from high to low addresses?  If
> > the allocation could be relied on to be top-down, then you wouldn't need
> > to adjust start at all, and it would return the highest available memory
> > in a natural way.
> 
> please check

It definitly gets us across that hump. Thanks.
> 
> [PATCH] x86, memblock: Fix early_node_mem with big reserved region.
> 
> Jeremy said Xen could reserve huge mem but still show as ram in e820.
> 
> early_node_mem could not find range because of start/end adjusting.
> 
> Let's use memblock_find_in_range instead ***_node. So get real top down in fallback path.
> 
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>

Tested-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

> 
> diff --git a/arch/x86/mm/numa_64.c b/arch/x86/mm/numa_64.c
> index 60f4985..7ffc9b7 100644
> --- a/arch/x86/mm/numa_64.c
> +++ b/arch/x86/mm/numa_64.c
> @@ -178,11 +178,8 @@ static void * __init early_node_mem(int nodeid, unsigned long start,
>  
>  	/* extend the search scope */
>  	end = max_pfn_mapped << PAGE_SHIFT;
> -	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
> -		start = MAX_DMA32_PFN<<PAGE_SHIFT;
> -	else
> -		start = MAX_DMA_PFN<<PAGE_SHIFT;
> -	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
> +	start = MAX_DMA_PFN << PAGE_SHIFT;
> +	mem = memblock_find_in_range(start, end, size, align);
>  	if (mem != MEMBLOCK_ERROR)
>  		return __va(mem);
>  

Re: early_node_mem()'s memory allocation policy

[Jeremy Fitzhardinge]

 On 10/26/2010 10:49 PM, Yinghai Lu wrote:
> please check
>
> [PATCH] x86, memblock: Fix early_node_mem with big reserved region.
>
> Jeremy said Xen could reserve huge mem but still show as ram in e820.
>
> early_node_mem could not find range because of start/end adjusting.
>
> Let's use memblock_find_in_range instead ***_node. So get real top down in fallback path.
>
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
>

Yes, that works.  Could you queue it for upstream?  Without this we see
Xen crashes with certain memory configurations.

Thanks,
    J

> diff --git a/arch/x86/mm/numa_64.c b/arch/x86/mm/numa_64.c
> index 60f4985..7ffc9b7 100644
> --- a/arch/x86/mm/numa_64.c
> +++ b/arch/x86/mm/numa_64.c
> @@ -178,11 +178,8 @@ static void * __init early_node_mem(int nodeid, unsigned long start,
>  
>  	/* extend the search scope */
>  	end = max_pfn_mapped << PAGE_SHIFT;
> -	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
> -		start = MAX_DMA32_PFN<<PAGE_SHIFT;
> -	else
> -		start = MAX_DMA_PFN<<PAGE_SHIFT;
> -	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
> +	start = MAX_DMA_PFN << PAGE_SHIFT;
> +	mem = memblock_find_in_range(start, end, size, align);
>  	if (mem != MEMBLOCK_ERROR)
>  		return __va(mem);
>  
>

[PATCH] x86, memblock: Fix early_node_mem with big reserved region.

[Yinghai Lu]

Jeremy said Xen could reserve huge mem but still show as ram in e820.
early_node_mem could not find range because of start/end adjusting.
and will go through fallback path. but fallback is still using 
memblock_x86_find_range_node(), and it is partially top-down because it
go through active_range entries from low to high.

Let's use memblock_find_in_range instead memblock_x86_find_range_node.
So get real top down in fallback path.

This is for 2.6.37.

And We may still need to make memblock_x86_find_range_node to do overall
top_down work.

Reported-by: Jeremy Fitzhardinge <jeremy@goop.org>
Tested-by: Jeremy Fitzhardinge <jeremy@goop.org>
Tested-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/mm/numa_64.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

Index: linux-2.6/arch/x86/mm/numa_64.c
===================================================================
--- linux-2.6.orig/arch/x86/mm/numa_64.c
+++ linux-2.6/arch/x86/mm/numa_64.c
@@ -178,11 +178,8 @@ static void * __init early_node_mem(int
 
 	/* extend the search scope */
 	end = max_pfn_mapped << PAGE_SHIFT;
-	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
-		start = MAX_DMA32_PFN<<PAGE_SHIFT;
-	else
-		start = MAX_DMA_PFN<<PAGE_SHIFT;
-	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
+	start = MAX_DMA_PFN << PAGE_SHIFT;
+	mem = memblock_find_in_range(start, end, size, align);
 	if (mem != MEMBLOCK_ERROR)
 		return __va(mem);
 

[tip:x86/urgent] x86, memblock: Fix early_node_mem with big reserved region.

[tip-bot for Yinghai Lu]

Commit-ID:  419db274bed4269f475a8e78cbe9c917192cfe8b
Gitweb:     http://git.kernel.org/tip/419db274bed4269f475a8e78cbe9c917192cfe8b
Author:     Yinghai Lu <yinghai@kernel.org>
AuthorDate: Thu, 28 Oct 2010 09:50:17 -0700
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Thu, 28 Oct 2010 15:52:36 -0700

x86, memblock: Fix early_node_mem with big reserved region.

Xen can reserve huge amounts of memory for pre-ballooning, but that
still shows as RAM in the e820 memory map.  early_node_mem could not
find range because of start/end adjusting, and will go through the
fallback path.  However, the fallback patch is still using
memblock_x86_find_range_node(), and it is partially top-down because
it go through active_range entries from low to high.

Let's use memblock_find_in_range instead memblock_x86_find_range_node.
So get real top down in fallback path.

We may still need to make memblock_x86_find_range_node to do overall
top_down work.

Reported-by: Jeremy Fitzhardinge <jeremy@goop.org>
Tested-by: Jeremy Fitzhardinge <jeremy@goop.org>
Tested-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
LKML-Reference: <4CC9A9C9.8020700@kernel.org>
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
---
 arch/x86/mm/numa_64.c |    7 ++-----
 1 files changed, 2 insertions(+), 5 deletions(-)

diff --git a/arch/x86/mm/numa_64.c b/arch/x86/mm/numa_64.c
index 60f4985..7ffc9b7 100644
--- a/arch/x86/mm/numa_64.c
+++ b/arch/x86/mm/numa_64.c
@@ -178,11 +178,8 @@ static void * __init early_node_mem(int nodeid, unsigned long start,
 
 	/* extend the search scope */
 	end = max_pfn_mapped << PAGE_SHIFT;
-	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
-		start = MAX_DMA32_PFN<<PAGE_SHIFT;
-	else
-		start = MAX_DMA_PFN<<PAGE_SHIFT;
-	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
+	start = MAX_DMA_PFN << PAGE_SHIFT;
+	mem = memblock_find_in_range(start, end, size, align);
 	if (mem != MEMBLOCK_ERROR)
 		return __va(mem);