[PATCH] arm64: Warn the user when a small VA_BITS value wastes memory

[PATCH] arm64: Warn the user when a small VA_BITS value wastes memory

[Marc Zyngier]

The memblock code ignores any memory that doesn't fit in the
linear mapping. In order to preserve the distance between two physical
memory locations and their mappings in the linear map, any hole between
two memory regions occupies the same space in the linear map.

On most systems, this is hardly a problem (the memory banks are close
together, and VA_BITS represents a large space compared to the available
memory *and* the potential gaps).

On NUMA systems, things are quite different: the gaps between the
memory nodes can be pretty large compared to the memory size itself,
and the range from memblock_start_of_DRAM() to memblock_end_of_DRAM()
can exceed the space described by VA_BITS.

Unfortunately, we're not very good at making this obvious to the user,
and on a D05 system (two sockets and 4 nodes with 64GB each)
accidentally configured with 39bit VA, we display something like this:

[    0.000000] NUMA: NODE_DATA [mem 0x1ffbffe100-0x1ffbffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x2febfc1100-0x2febfc2fff]
[    0.000000] NUMA: Initmem setup node 2 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x2febfbf200-0x2febfc10ff]
[    0.000000] NUMA: NODE_DATA(2) on node 1
[    0.000000] NUMA: Initmem setup node 3 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x2febfbd300-0x2febfbf1ff]
[    0.000000] NUMA: NODE_DATA(3) on node 1

which isn't very explicit, and doesn't tell the user why 128GB
have suddently disappeared.

Let's add a warning message telling the user that memory has been
truncated, and offer a potential solution (bumping VA_BITS up).

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/mm/init.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 095540667f0f..cd5da059de85 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -283,6 +283,9 @@ void __init arm64_memblock_init(void)
 	memstart_addr = round_down(memblock_start_of_DRAM(),
 				   ARM64_MEMSTART_ALIGN);
 
+	if ((memblock_end_of_DRAM() - memstart_addr) > linear_region_size)
+		pr_warn("Memory doesn't fit in the linear mapping, VA_BITS too small\n");
+
 	/*
 	 * Remove the memory that we will not be able to cover with the
 	 * linear mapping. Take care not to clip the kernel which may be
-- 
2.29.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] arm64: Warn the user when a small VA_BITS value wastes memory

[Mark Rutland]

On Tue, Dec 15, 2020 at 03:29:18PM +0000, Marc Zyngier wrote:
> The memblock code ignores any memory that doesn't fit in the
> linear mapping. In order to preserve the distance between two physical
> memory locations and their mappings in the linear map, any hole between
> two memory regions occupies the same space in the linear map.
> 
> On most systems, this is hardly a problem (the memory banks are close
> together, and VA_BITS represents a large space compared to the available
> memory *and* the potential gaps).
> 
> On NUMA systems, things are quite different: the gaps between the
> memory nodes can be pretty large compared to the memory size itself,
> and the range from memblock_start_of_DRAM() to memblock_end_of_DRAM()
> can exceed the space described by VA_BITS.
> 
> Unfortunately, we're not very good at making this obvious to the user,
> and on a D05 system (two sockets and 4 nodes with 64GB each)
> accidentally configured with 39bit VA, we display something like this:
> 
> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffe100-0x1ffbffffff]
> [    0.000000] NUMA: NODE_DATA [mem 0x2febfc1100-0x2febfc2fff]
> [    0.000000] NUMA: Initmem setup node 2 [<memory-less node>]
> [    0.000000] NUMA: NODE_DATA [mem 0x2febfbf200-0x2febfc10ff]
> [    0.000000] NUMA: NODE_DATA(2) on node 1
> [    0.000000] NUMA: Initmem setup node 3 [<memory-less node>]
> [    0.000000] NUMA: NODE_DATA [mem 0x2febfbd300-0x2febfbf1ff]
> [    0.000000] NUMA: NODE_DATA(3) on node 1
> 
> which isn't very explicit, and doesn't tell the user why 128GB
> have suddently disappeared.
> 
> Let's add a warning message telling the user that memory has been
> truncated, and offer a potential solution (bumping VA_BITS up).
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/mm/init.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index 095540667f0f..cd5da059de85 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -283,6 +283,9 @@ void __init arm64_memblock_init(void)
>  	memstart_addr = round_down(memblock_start_of_DRAM(),
>  				   ARM64_MEMSTART_ALIGN);
>  
> +	if ((memblock_end_of_DRAM() - memstart_addr) > linear_region_size)
> +		pr_warn("Memory doesn't fit in the linear mapping, VA_BITS too small\n");


Hmm... we might want to restructure this code a bit, since we have a few
checks that are /almost/ the same, and we could probably place all the
memory clipping under this new condition. That said, after having a go,
I couldn't find a neat solution.

FWIW, for this as-is:

Acked-by: Mark Rutland <mark.rutland@arm.com>

Mark.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] arm64: Warn the user when a small VA_BITS value wastes memory

[Catalin Marinas]

On Tue, 15 Dec 2020 15:29:18 +0000, Marc Zyngier wrote:
> The memblock code ignores any memory that doesn't fit in the
> linear mapping. In order to preserve the distance between two physical
> memory locations and their mappings in the linear map, any hole between
> two memory regions occupies the same space in the linear map.
> 
> On most systems, this is hardly a problem (the memory banks are close
> together, and VA_BITS represents a large space compared to the available
> memory *and* the potential gaps).
> 
> [...]

Applied to arm64 (for-next/fixes), thanks!

[1/1] arm64: Warn the user when a small VA_BITS value wastes memory
      https://git.kernel.org/arm64/c/31f80a4e9603

-- 
Catalin


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel