[PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Chen Feng]

We can reduce the memory allocated at mem-map
by flatmem.

currently, the default memory-model in arm64 is
sparse memory. The mem-map array is not freed in
this scene. If the physical address is too long,
it will reserved too much memory for the mem-map
array.

Signed-off-by: Chen Feng <puck.chen@hisilicon.com>
Signed-off-by: Fu Jun <oliver.fu@hisilicon.com>
---
 arch/arm64/Kconfig | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 4f43622..c18930d 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -559,6 +559,9 @@ config ARCH_SPARSEMEM_ENABLE
 	def_bool y
 	select SPARSEMEM_VMEMMAP_ENABLE
 
+config ARCH_FLATMEM_ENABLE
+	def_bool y
+
 config ARCH_SPARSEMEM_DEFAULT
 	def_bool ARCH_SPARSEMEM_ENABLE
 
-- 
1.9.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 2/2] arm64: mm: make pfn always valid with flat memory

[Chen Feng]

Make the pfn always valid when using flat memory.
If the reserved memory is not align to memblock-size,
there will be holes in zone.

This patch makes the memory in buddy always in the
array of mem-map.

Signed-off-by: Chen Feng <puck.chen@hisilicon.com>
Signed-off-by: Fu Jun <oliver.fu@hisilicon.com>
---
 arch/arm64/mm/init.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index ea989d8..0e1d5b7 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -306,7 +306,8 @@ static void __init free_unused_memmap(void)
 	struct memblock_region *reg;
 
 	for_each_memblock(memory, reg) {
-		start = __phys_to_pfn(reg->base);
+		start = round_down(__phys_to_pfn(reg->base),
+				   MAX_ORDER_NR_PAGES);
 
 #ifdef CONFIG_SPARSEMEM
 		/*
@@ -327,8 +328,8 @@ static void __init free_unused_memmap(void)
 		 * memmap entries are valid from the bank end aligned to
 		 * MAX_ORDER_NR_PAGES.
 		 */
-		prev_end = ALIGN(__phys_to_pfn(reg->base + reg->size),
-				 MAX_ORDER_NR_PAGES);
+		prev_end = round_up(__phys_to_pfn(reg->base + reg->size),
+				    MAX_ORDER_NR_PAGES);
 	}
 
 #ifdef CONFIG_SPARSEMEM
-- 
1.9.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 2/2] arm64: mm: make pfn always valid with flat memory

[Chen Feng]

add Mel Gorman

On 2016/4/5 16:22, Chen Feng wrote:
> Make the pfn always valid when using flat memory.
> If the reserved memory is not align to memblock-size,
> there will be holes in zone.
> 
> This patch makes the memory in buddy always in the
> array of mem-map.
> 
> Signed-off-by: Chen Feng <puck.chen@hisilicon.com>
> Signed-off-by: Fu Jun <oliver.fu@hisilicon.com>
> ---
>  arch/arm64/mm/init.c | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index ea989d8..0e1d5b7 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -306,7 +306,8 @@ static void __init free_unused_memmap(void)
>  	struct memblock_region *reg;
>  
>  	for_each_memblock(memory, reg) {
> -		start = __phys_to_pfn(reg->base);
> +		start = round_down(__phys_to_pfn(reg->base),
> +				   MAX_ORDER_NR_PAGES);
>  
>  #ifdef CONFIG_SPARSEMEM
>  		/*
> @@ -327,8 +328,8 @@ static void __init free_unused_memmap(void)
>  		 * memmap entries are valid from the bank end aligned to
>  		 * MAX_ORDER_NR_PAGES.
>  		 */
> -		prev_end = ALIGN(__phys_to_pfn(reg->base + reg->size),
> -				 MAX_ORDER_NR_PAGES);
> +		prev_end = round_up(__phys_to_pfn(reg->base + reg->size),
> +				    MAX_ORDER_NR_PAGES);
>  	}
>  
>  #ifdef CONFIG_SPARSEMEM
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 2/2] arm64: mm: make pfn always valid with flat memory

[Xishi Qiu]

On 2016/4/5 16:22, Chen Feng wrote:

> Make the pfn always valid when using flat memory.
> If the reserved memory is not align to memblock-size,
> there will be holes in zone.
> 
> This patch makes the memory in buddy always in the
> array of mem-map.
> 
> Signed-off-by: Chen Feng <puck.chen@hisilicon.com>
> Signed-off-by: Fu Jun <oliver.fu@hisilicon.com>
> ---
>  arch/arm64/mm/init.c | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index ea989d8..0e1d5b7 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -306,7 +306,8 @@ static void __init free_unused_memmap(void)

How about let free_unused_memmap() support for CONFIG_SPARSEMEM_VMEMMAP?

Thanks,
Xishi Qiu

>  	struct memblock_region *reg;
>  
>  	for_each_memblock(memory, reg) {
> -		start = __phys_to_pfn(reg->base);
> +		start = round_down(__phys_to_pfn(reg->base),
> +				   MAX_ORDER_NR_PAGES);
>  
>  #ifdef CONFIG_SPARSEMEM
>  		/*
> @@ -327,8 +328,8 @@ static void __init free_unused_memmap(void)
>  		 * memmap entries are valid from the bank end aligned to
>  		 * MAX_ORDER_NR_PAGES.
>  		 */
> -		prev_end = ALIGN(__phys_to_pfn(reg->base + reg->size),
> -				 MAX_ORDER_NR_PAGES);
> +		prev_end = round_up(__phys_to_pfn(reg->base + reg->size),
> +				    MAX_ORDER_NR_PAGES);
>  	}
>  
>  #ifdef CONFIG_SPARSEMEM



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 2/2] arm64: mm: make pfn always valid with flat memory

[Catalin Marinas]

On Mon, Apr 11, 2016 at 07:08:38PM +0800, Xishi Qiu wrote:
> On 2016/4/5 16:22, Chen Feng wrote:
> 
> > Make the pfn always valid when using flat memory.
> > If the reserved memory is not align to memblock-size,
> > there will be holes in zone.
> > 
> > This patch makes the memory in buddy always in the
> > array of mem-map.
> > 
> > Signed-off-by: Chen Feng <puck.chen@hisilicon.com>
> > Signed-off-by: Fu Jun <oliver.fu@hisilicon.com>
> > ---
> >  arch/arm64/mm/init.c | 7 ++++---
> >  1 file changed, 4 insertions(+), 3 deletions(-)
> > 
> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > index ea989d8..0e1d5b7 100644
> > --- a/arch/arm64/mm/init.c
> > +++ b/arch/arm64/mm/init.c
> > @@ -306,7 +306,8 @@ static void __init free_unused_memmap(void)
> 
> How about let free_unused_memmap() support for CONFIG_SPARSEMEM_VMEMMAP?

We would need extra care to check that the memmap was actually allocated
in the first place.

-- 
Catalin

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Chen Feng]

add Mel Gorman

On 2016/4/5 16:22, Chen Feng wrote:
> We can reduce the memory allocated at mem-map
> by flatmem.
> 
> currently, the default memory-model in arm64 is
> sparse memory. The mem-map array is not freed in
> this scene. If the physical address is too long,
> it will reserved too much memory for the mem-map
> array.
> 
> Signed-off-by: Chen Feng <puck.chen@hisilicon.com>
> Signed-off-by: Fu Jun <oliver.fu@hisilicon.com>
> ---
>  arch/arm64/Kconfig | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 4f43622..c18930d 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -559,6 +559,9 @@ config ARCH_SPARSEMEM_ENABLE
>  	def_bool y
>  	select SPARSEMEM_VMEMMAP_ENABLE
>  
> +config ARCH_FLATMEM_ENABLE
> +	def_bool y
> +
>  config ARCH_SPARSEMEM_DEFAULT
>  	def_bool ARCH_SPARSEMEM_ENABLE
>  
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Will Deacon]

On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
> We can reduce the memory allocated at mem-map
> by flatmem.
> 
> currently, the default memory-model in arm64 is
> sparse memory. The mem-map array is not freed in
> this scene. If the physical address is too long,
> it will reserved too much memory for the mem-map
> array.

Can you elaborate a bit more on this, please? We use the vmemmap, so any
spaces between memory banks only burns up virtual space. What exactly is
the problem you're seeing that makes you want to use flatmem (which is
probably unsuitable for the majority of arm64 machines).

Will

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Chen Feng]

Hi will,
Thanks for review.

On 2016/4/7 22:21, Will Deacon wrote:
> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>> We can reduce the memory allocated at mem-map
>> by flatmem.
>>
>> currently, the default memory-model in arm64 is
>> sparse memory. The mem-map array is not freed in
>> this scene. If the physical address is too long,
>> it will reserved too much memory for the mem-map
>> array.
> 
> Can you elaborate a bit more on this, please? We use the vmemmap, so any
> spaces between memory banks only burns up virtual space. What exactly is
> the problem you're seeing that makes you want to use flatmem (which is
> probably unsuitable for the majority of arm64 machines).
> 
The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
is not freed.

take a look at here:
arm64/mm/init.c
void __init mem_init(void)
{
#ifndef CONFIG_SPARSEMEM_VMEMMAP
	free_unused_memmap();
#endif
}

Memory layout (3GB)

 0             1.5G    2G             3.5G            4G
 |              |      |               |              |
 +--------------+------+---------------+--------------+
 |    MEM       | hole |     MEM       |   IO (regs)  |
 +--------------+------+---------------+--------------+


Memory layout (4GB)

 0                                    3.5G            4G    4.5G
 |                                     |              |       |
 +-------------------------------------+--------------+-------+
 |                   MEM               |   IO (regs)  |  MEM  |
 +-------------------------------------+--------------+-------+

Currently, the sparse memory section is 1GB.

3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.

This will alloc 1G/4K * (struct page) memory for mem_map array.

We want to use flat-mem to reduce the alloced mem_map.

I don't know why you tell us the flatmem is unsuitable for the
majority of arm64 machines. Can tell us the reason of it?

And we are not going to limit the memdel in arm64, we just want to
make the flat-mem is an optional item in arm64.


puck,


> Will
> 
> .
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Ard Biesheuvel]

On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
> Hi will,
> Thanks for review.
>
> On 2016/4/7 22:21, Will Deacon wrote:
>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>> We can reduce the memory allocated at mem-map
>>> by flatmem.
>>>
>>> currently, the default memory-model in arm64 is
>>> sparse memory. The mem-map array is not freed in
>>> this scene. If the physical address is too long,
>>> it will reserved too much memory for the mem-map
>>> array.
>>
>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>> spaces between memory banks only burns up virtual space. What exactly is
>> the problem you're seeing that makes you want to use flatmem (which is
>> probably unsuitable for the majority of arm64 machines).
>>
> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
> is not freed.
>
> take a look at here:
> arm64/mm/init.c
> void __init mem_init(void)
> {
> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>         free_unused_memmap();
> #endif
> }
>
> Memory layout (3GB)
>
>  0             1.5G    2G             3.5G            4G
>  |              |      |               |              |
>  +--------------+------+---------------+--------------+
>  |    MEM       | hole |     MEM       |   IO (regs)  |
>  +--------------+------+---------------+--------------+
>
>
> Memory layout (4GB)
>
>  0                                    3.5G            4G    4.5G
>  |                                     |              |       |
>  +-------------------------------------+--------------+-------+
>  |                   MEM               |   IO (regs)  |  MEM  |
>  +-------------------------------------+--------------+-------+
>
> Currently, the sparse memory section is 1GB.
>
> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>
> This will alloc 1G/4K * (struct page) memory for mem_map array.
>

No, this is incorrect. Sparsemem vmemmap only allocates struct pages
for memory regions that are actually populated.

For instance, on the Foundation model with 4 GB of memory, you may see
something like this in the boot log

[    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
(     8 GB maximum)
[    0.000000]               0xffffffbdc0000000 - 0xffffffbde2000000
(   544 MB actual)

but in reality, only the following regions have been allocated

---[ vmemmap start ]---
0xffffffbdc0000000-0xffffffbdc2000000          32M       RW NX SHD AF
      BLK UXN MEM/NORMAL
0xffffffbde0000000-0xffffffbde2000000          32M       RW NX SHD AF
      BLK UXN MEM/NORMAL
---[ vmemmap end ]---

so only 64 MB is used to back 4 GB of RAM with struct pages, which is
minimal. Moving to flatmem will not reduce the memory footprint at
all.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Chen Feng]

Hi Ard,

On 2016/4/11 15:35, Ard Biesheuvel wrote:
> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
>> Hi will,
>> Thanks for review.
>>
>> On 2016/4/7 22:21, Will Deacon wrote:
>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>> We can reduce the memory allocated at mem-map
>>>> by flatmem.
>>>>
>>>> currently, the default memory-model in arm64 is
>>>> sparse memory. The mem-map array is not freed in
>>>> this scene. If the physical address is too long,
>>>> it will reserved too much memory for the mem-map
>>>> array.
>>>
>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>> spaces between memory banks only burns up virtual space. What exactly is
>>> the problem you're seeing that makes you want to use flatmem (which is
>>> probably unsuitable for the majority of arm64 machines).
>>>
>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>> is not freed.
>>
>> take a look at here:
>> arm64/mm/init.c
>> void __init mem_init(void)
>> {
>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>         free_unused_memmap();
>> #endif
>> }
>>
>> Memory layout (3GB)
>>
>>  0             1.5G    2G             3.5G            4G
>>  |              |      |               |              |
>>  +--------------+------+---------------+--------------+
>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>  +--------------+------+---------------+--------------+
>>
>>
>> Memory layout (4GB)
>>
>>  0                                    3.5G            4G    4.5G
>>  |                                     |              |       |
>>  +-------------------------------------+--------------+-------+
>>  |                   MEM               |   IO (regs)  |  MEM  |
>>  +-------------------------------------+--------------+-------+
>>
>> Currently, the sparse memory section is 1GB.
>>
>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>
>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>
> 
> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
> for memory regions that are actually populated.
> 
> For instance, on the Foundation model with 4 GB of memory, you may see
> something like this in the boot log
> 
> [    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
> (     8 GB maximum)
> [    0.000000]               0xffffffbdc0000000 - 0xffffffbde2000000
> (   544 MB actual)
> 
> but in reality, only the following regions have been allocated
> 
> ---[ vmemmap start ]---
> 0xffffffbdc0000000-0xffffffbdc2000000          32M       RW NX SHD AF
>       BLK UXN MEM/NORMAL
> 0xffffffbde0000000-0xffffffbde2000000          32M       RW NX SHD AF
>       BLK UXN MEM/NORMAL
> ---[ vmemmap end ]---
> 
> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
> minimal. Moving to flatmem will not reduce the memory footprint at
> all.

Yes,but the populate is section, which is 1GB. Take a look at the above
memory layout.

The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.

The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>  0             1.5G    2G             3.5G            4G
>>  |              |      |               |              |
>>  +--------------+------+---------------+--------------+
>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>  +--------------+------+---------------+--------------+
The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.

We want free the the mem-map array. With flat-mem we can work with this scene very well.

Thanks,


> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 
> .
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Ard Biesheuvel]

On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
> Hi Ard,
>
> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
>>> Hi will,
>>> Thanks for review.
>>>
>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>> We can reduce the memory allocated at mem-map
>>>>> by flatmem.
>>>>>
>>>>> currently, the default memory-model in arm64 is
>>>>> sparse memory. The mem-map array is not freed in
>>>>> this scene. If the physical address is too long,
>>>>> it will reserved too much memory for the mem-map
>>>>> array.
>>>>
>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>> probably unsuitable for the majority of arm64 machines).
>>>>
>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>> is not freed.
>>>
>>> take a look at here:
>>> arm64/mm/init.c
>>> void __init mem_init(void)
>>> {
>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>         free_unused_memmap();
>>> #endif
>>> }
>>>
>>> Memory layout (3GB)
>>>
>>>  0             1.5G    2G             3.5G            4G
>>>  |              |      |               |              |
>>>  +--------------+------+---------------+--------------+
>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>  +--------------+------+---------------+--------------+
>>>
>>>
>>> Memory layout (4GB)
>>>
>>>  0                                    3.5G            4G    4.5G
>>>  |                                     |              |       |
>>>  +-------------------------------------+--------------+-------+
>>>  |                   MEM               |   IO (regs)  |  MEM  |
>>>  +-------------------------------------+--------------+-------+
>>>
>>> Currently, the sparse memory section is 1GB.
>>>
>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>
>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>
>>
>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>> for memory regions that are actually populated.
>>
>> For instance, on the Foundation model with 4 GB of memory, you may see
>> something like this in the boot log
>>
>> [    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>> (     8 GB maximum)
>> [    0.000000]               0xffffffbdc0000000 - 0xffffffbde2000000
>> (   544 MB actual)
>>
>> but in reality, only the following regions have been allocated
>>
>> ---[ vmemmap start ]---
>> 0xffffffbdc0000000-0xffffffbdc2000000          32M       RW NX SHD AF
>>       BLK UXN MEM/NORMAL
>> 0xffffffbde0000000-0xffffffbde2000000          32M       RW NX SHD AF
>>       BLK UXN MEM/NORMAL
>> ---[ vmemmap end ]---
>>
>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>> minimal. Moving to flatmem will not reduce the memory footprint at
>> all.
>
> Yes,but the populate is section, which is 1GB. Take a look at the above
> memory layout.
>
> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>
> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>  0             1.5G    2G             3.5G            4G
>>>  |              |      |               |              |
>>>  +--------------+------+---------------+--------------+
>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>  +--------------+------+---------------+--------------+
> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>

No, it is not. It may be covered by a section, but that does not mean
sparsemem vmemmap will actually allocate backing for it. The
granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
due to the fact that the backing is performed at PMD granularity.

Please, could you share the contents of the vmemmap section in
/sys/kernel/debug/kernel_page_tables of your system running with
sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Chen Feng]

Hi Ard,

On 2016/4/11 16:00, Ard Biesheuvel wrote:
> On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
>> Hi Ard,
>>
>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>> Hi will,
>>>> Thanks for review.
>>>>
>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>> We can reduce the memory allocated at mem-map
>>>>>> by flatmem.
>>>>>>
>>>>>> currently, the default memory-model in arm64 is
>>>>>> sparse memory. The mem-map array is not freed in
>>>>>> this scene. If the physical address is too long,
>>>>>> it will reserved too much memory for the mem-map
>>>>>> array.
>>>>>
>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>
>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>> is not freed.
>>>>
>>>> take a look at here:
>>>> arm64/mm/init.c
>>>> void __init mem_init(void)
>>>> {
>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>>         free_unused_memmap();
>>>> #endif
>>>> }
>>>>
>>>> Memory layout (3GB)
>>>>
>>>>  0             1.5G    2G             3.5G            4G
>>>>  |              |      |               |              |
>>>>  +--------------+------+---------------+--------------+
>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>  +--------------+------+---------------+--------------+
>>>>
>>>>
>>>> Memory layout (4GB)
>>>>
>>>>  0                                    3.5G            4G    4.5G
>>>>  |                                     |              |       |
>>>>  +-------------------------------------+--------------+-------+
>>>>  |                   MEM               |   IO (regs)  |  MEM  |
>>>>  +-------------------------------------+--------------+-------+
>>>>
>>>> Currently, the sparse memory section is 1GB.
>>>>
>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>
>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>
>>>
>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>> for memory regions that are actually populated.
>>>
>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>> something like this in the boot log
>>>
>>> [    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>> (     8 GB maximum)
>>> [    0.000000]               0xffffffbdc0000000 - 0xffffffbde2000000
>>> (   544 MB actual)
>>>
>>> but in reality, only the following regions have been allocated
>>>
>>> ---[ vmemmap start ]---
>>> 0xffffffbdc0000000-0xffffffbdc2000000          32M       RW NX SHD AF
>>>       BLK UXN MEM/NORMAL
>>> 0xffffffbde0000000-0xffffffbde2000000          32M       RW NX SHD AF
>>>       BLK UXN MEM/NORMAL
>>> ---[ vmemmap end ]---
>>>
>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>> all.
>>
>> Yes,but the populate is section, which is 1GB. Take a look at the above
>> memory layout.
>>
>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>
>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>>  0             1.5G    2G             3.5G            4G
>>>>  |              |      |               |              |
>>>>  +--------------+------+---------------+--------------+
>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>  +--------------+------+---------------+--------------+
>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>
> 
> No, it is not. It may be covered by a section, but that does not mean
> sparsemem vmemmap will actually allocate backing for it. The
> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
> due to the fact that the backing is performed at PMD granularity.
> 
> Please, could you share the contents of the vmemmap section in
> /sys/kernel/debug/kernel_page_tables of your system running with
> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>

Please see the pg-tables below.


With sparse and vmemmap enable.

---[ vmemmap start ]---
0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
---[ vmemmap end ]---


The board is 4GB, and the memap is 70MB
1G memory --- 14MB mem_map array.
So the 4GB has 5 sections, which used 5 * 14MB memory.






> .
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Ard Biesheuvel]

On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
> Hi Ard,
>
> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>> On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
>>> Hi Ard,
>>>
>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>>> Hi will,
>>>>> Thanks for review.
>>>>>
>>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>>> We can reduce the memory allocated at mem-map
>>>>>>> by flatmem.
>>>>>>>
>>>>>>> currently, the default memory-model in arm64 is
>>>>>>> sparse memory. The mem-map array is not freed in
>>>>>>> this scene. If the physical address is too long,
>>>>>>> it will reserved too much memory for the mem-map
>>>>>>> array.
>>>>>>
>>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>>
>>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>>> is not freed.
>>>>>
>>>>> take a look at here:
>>>>> arm64/mm/init.c
>>>>> void __init mem_init(void)
>>>>> {
>>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>>>         free_unused_memmap();
>>>>> #endif
>>>>> }
>>>>>
>>>>> Memory layout (3GB)
>>>>>
>>>>>  0             1.5G    2G             3.5G            4G
>>>>>  |              |      |               |              |
>>>>>  +--------------+------+---------------+--------------+
>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>  +--------------+------+---------------+--------------+
>>>>>
>>>>>
>>>>> Memory layout (4GB)
>>>>>
>>>>>  0                                    3.5G            4G    4.5G
>>>>>  |                                     |              |       |
>>>>>  +-------------------------------------+--------------+-------+
>>>>>  |                   MEM               |   IO (regs)  |  MEM  |
>>>>>  +-------------------------------------+--------------+-------+
>>>>>
>>>>> Currently, the sparse memory section is 1GB.
>>>>>
>>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>>
>>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>>
>>>>
>>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>>> for memory regions that are actually populated.
>>>>
>>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>>> something like this in the boot log
>>>>
>>>> [    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>>> (     8 GB maximum)
>>>> [    0.000000]               0xffffffbdc0000000 - 0xffffffbde2000000
>>>> (   544 MB actual)
>>>>
>>>> but in reality, only the following regions have been allocated
>>>>
>>>> ---[ vmemmap start ]---
>>>> 0xffffffbdc0000000-0xffffffbdc2000000          32M       RW NX SHD AF
>>>>       BLK UXN MEM/NORMAL
>>>> 0xffffffbde0000000-0xffffffbde2000000          32M       RW NX SHD AF
>>>>       BLK UXN MEM/NORMAL
>>>> ---[ vmemmap end ]---
>>>>
>>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>>> all.
>>>
>>> Yes,but the populate is section, which is 1GB. Take a look at the above
>>> memory layout.
>>>
>>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>>
>>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>>>  0             1.5G    2G             3.5G            4G
>>>>>  |              |      |               |              |
>>>>>  +--------------+------+---------------+--------------+
>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>  +--------------+------+---------------+--------------+
>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>
>>
>> No, it is not. It may be covered by a section, but that does not mean
>> sparsemem vmemmap will actually allocate backing for it. The
>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>> due to the fact that the backing is performed at PMD granularity.
>>
>> Please, could you share the contents of the vmemmap section in
>> /sys/kernel/debug/kernel_page_tables of your system running with
>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>
>
> Please see the pg-tables below.
>
>
> With sparse and vmemmap enable.
>
> ---[ vmemmap start ]---
> 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
> ---[ vmemmap end ]---
>

OK, I see what you mean now. Sorry for taking so long to catch up.

> The board is 4GB, and the memap is 70MB
> 1G memory --- 14MB mem_map array.

No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
pages assuming sizeof(struct page) == 64

So you are losing 6 MB to rounding here, which I agree is significant.
I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
on 4k pages kernels, but perhaps we're better off asking the opinion
of the other cc'ees.

Thanks,
Ard.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Will Deacon]

On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
> On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
> > Please see the pg-tables below.
> >
> >
> > With sparse and vmemmap enable.
> >
> > ---[ vmemmap start ]---
> > 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
> > ---[ vmemmap end ]---
> >
> 
> OK, I see what you mean now. Sorry for taking so long to catch up.
> 
> > The board is 4GB, and the memap is 70MB
> > 1G memory --- 14MB mem_map array.
> 
> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
> pages assuming sizeof(struct page) == 64
> 
> So you are losing 6 MB to rounding here, which I agree is significant.
> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
> on 4k pages kernels, but perhaps we're better off asking the opinion
> of the other cc'ees.

You need to be really careful making SECTION_SIZE_BITS smaller because
it has a direct correlation on the use of page->flags and you can end up
running out of bits fairly easily.

Will

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Chen Feng]

Hi Will,

On 2016/4/11 18:40, Will Deacon wrote:
> On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
>> On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
>>> Please see the pg-tables below.
>>>
>>>
>>> With sparse and vmemmap enable.
>>>
>>> ---[ vmemmap start ]---
>>> 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
>>> ---[ vmemmap end ]---
>>>
>>
>> OK, I see what you mean now. Sorry for taking so long to catch up.
>>
>>> The board is 4GB, and the memap is 70MB
>>> 1G memory --- 14MB mem_map array.
>>
>> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
>> pages assuming sizeof(struct page) == 64
>>
>> So you are losing 6 MB to rounding here, which I agree is significant.
>> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
>> on 4k pages kernels, but perhaps we're better off asking the opinion
>> of the other cc'ees.
> 
> You need to be really careful making SECTION_SIZE_BITS smaller because
> it has a direct correlation on the use of page->flags and you can end up
> running out of bits fairly easily.

Yes, making SECTION_SIZE_BITS smaller can solve the current situation.

But if the phys-addr is 64GB, but only 4GB ddr is the valid address. And the

holes are not always 512MB.

But, can you tell us why *smaller SIZE makes running out of bits fairly easily*?

And how about the flat-mem model?

> 
> Will
> 
> .
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Laura Abbott]

On 04/11/2016 03:57 AM, Chen Feng wrote:
> Hi Will,
>
> On 2016/4/11 18:40, Will Deacon wrote:
>> On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
>>> On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>> Please see the pg-tables below.
>>>>
>>>>
>>>> With sparse and vmemmap enable.
>>>>
>>>> ---[ vmemmap start ]---
>>>> 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
>>>> ---[ vmemmap end ]---
>>>>
>>>
>>> OK, I see what you mean now. Sorry for taking so long to catch up.
>>>
>>>> The board is 4GB, and the memap is 70MB
>>>> 1G memory --- 14MB mem_map array.
>>>
>>> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
>>> pages assuming sizeof(struct page) == 64
>>>
>>> So you are losing 6 MB to rounding here, which I agree is significant.
>>> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
>>> on 4k pages kernels, but perhaps we're better off asking the opinion
>>> of the other cc'ees.
>>
>> You need to be really careful making SECTION_SIZE_BITS smaller because
>> it has a direct correlation on the use of page->flags and you can end up
>> running out of bits fairly easily.
>
> Yes, making SECTION_SIZE_BITS smaller can solve the current situation.
>
> But if the phys-addr is 64GB, but only 4GB ddr is the valid address. And the
>
> holes are not always 512MB.
>
> But, can you tell us why *smaller SIZE makes running out of bits fairly easily*?
>

Think about page tables and TLB pressure. A larger page size can cover the
same memory area with fewer page table entries. The same type of logic applies
to memory sections here as well. If the section size is smaller, you need
more bits to represent the number of sections used. page->flags is a long

In include/linux/mm.h

/* Page flags: | [SECTION] | [NODE] | ZONE | [LAST_CPUPID] | ... | FLAGS | */

and

#if SECTIONS_WIDTH+NODES_WIDTH+ZONES_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS
#error SECTIONS_WIDTH+NODES_WIDTH+ZONES_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS
#endif

So it's a trade off of what can be encoded in an unsigned long.

We're hitting the upper bound on zones as well (see 033fbae988fc 'mm:
ZONE_DEVICE for "device memory"')


> And how about the flat-mem model?
>
>>
>> Will
>>
>> .
>>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Catalin Marinas]

On Mon, Apr 11, 2016 at 11:40:13AM +0100, Will Deacon wrote:
> On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
> > On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
> > > Please see the pg-tables below.
> > >
> > >
> > > With sparse and vmemmap enable.
> > >
> > > ---[ vmemmap start ]---
> > > 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
> > > ---[ vmemmap end ]---
> > >
> > 
> > OK, I see what you mean now. Sorry for taking so long to catch up.
> > 
> > > The board is 4GB, and the memap is 70MB
> > > 1G memory --- 14MB mem_map array.
> > 
> > No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
> > pages assuming sizeof(struct page) == 64
> > 
> > So you are losing 6 MB to rounding here, which I agree is significant.
> > I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
> > on 4k pages kernels, but perhaps we're better off asking the opinion
> > of the other cc'ees.
> 
> You need to be really careful making SECTION_SIZE_BITS smaller because
> it has a direct correlation on the use of page->flags and you can end up
> running out of bits fairly easily.

With SPARSEMEM_VMEMMAP, SECTION_SIZE_BITS no longer affect the page
flags since we no longer need to encode the section number in
page->flags.

-- 
Catalin

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Catalin Marinas]

On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
> On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
> > On 2016/4/11 16:00, Ard Biesheuvel wrote:
> >> On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
> >>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
> >>>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
> >>>>>  0             1.5G    2G             3.5G            4G
> >>>>>  |              |      |               |              |
> >>>>>  +--------------+------+---------------+--------------+
> >>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
> >>>>>  +--------------+------+---------------+--------------+
> >>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
> >>>
> >>
> >> No, it is not. It may be covered by a section, but that does not mean
> >> sparsemem vmemmap will actually allocate backing for it. The
> >> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
> >> due to the fact that the backing is performed at PMD granularity.
> >>
> >> Please, could you share the contents of the vmemmap section in
> >> /sys/kernel/debug/kernel_page_tables of your system running with
> >> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
> >
> > Please see the pg-tables below.
> >
> > With sparse and vmemmap enable.
> >
> > ---[ vmemmap start ]---
> > 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
> > ---[ vmemmap end ]---
[...]
> > The board is 4GB, and the memap is 70MB
> > 1G memory --- 14MB mem_map array.
> 
> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
> pages assuming sizeof(struct page) == 64
> 
> So you are losing 6 MB to rounding here, which I agree is significant.
> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
> on 4k pages kernels, but perhaps we're better off asking the opinion
> of the other cc'ees.

IIRC, SECTION_SIZE_BITS was chosen to be the maximum sane value we were
thinking of at the time, assuming that 1GB RAM alignment to be fairly
normal. For the !SPARSEMEM_VMEMMAP case, we should probably be fine with
29 but, as Will said, we need to be careful with the page flags. At a
quick look, we have 25 page flags, 2 bits per zone, NUMA nodes and (48 -
section_size_bits) for the section width. We also need to take into
account 4 more bits for 52-bit PA support (ARMv8.2). So, without NUMA
nodes, we are currently at 49 bits used in page->flags.

For the SPARSEMEM_VMEMMAP case, we can decrease the SECTION_SIZE_BITS in
the MAX_ORDER limit.

An alternative would be to free the vmemmap holes later (but still keep
the vmemmap mapping alias). Yet another option would be to change the
sparse_mem_map_populate() logic get the actual section end rather than
always assuming PAGES_PER_SECTION. But I don't think any of these are
worth if we can safely reduce SECTION_SIZE_BITS.

-- 
Catalin

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Chen Feng]

Hi Catalin,

Thanks for your reply.
On 2016/4/12 22:59, Catalin Marinas wrote:
> On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
>> On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
>>> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>>>> On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>>>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>>>>>  0             1.5G    2G             3.5G            4G
>>>>>>>  |              |      |               |              |
>>>>>>>  +--------------+------+---------------+--------------+
>>>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>>>  +--------------+------+---------------+--------------+
>>>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>>>
>>>>
>>>> No, it is not. It may be covered by a section, but that does not mean
>>>> sparsemem vmemmap will actually allocate backing for it. The
>>>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>>>> due to the fact that the backing is performed at PMD granularity.
>>>>
>>>> Please, could you share the contents of the vmemmap section in
>>>> /sys/kernel/debug/kernel_page_tables of your system running with
>>>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>>
>>> Please see the pg-tables below.
>>>
>>> With sparse and vmemmap enable.
>>>
>>> ---[ vmemmap start ]---
>>> 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
>>> ---[ vmemmap end ]---
> [...]
>>> The board is 4GB, and the memap is 70MB
>>> 1G memory --- 14MB mem_map array.
>>
>> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
>> pages assuming sizeof(struct page) == 64
>>
>> So you are losing 6 MB to rounding here, which I agree is significant.
>> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
>> on 4k pages kernels, but perhaps we're better off asking the opinion
>> of the other cc'ees.
> 
> IIRC, SECTION_SIZE_BITS was chosen to be the maximum sane value we were
> thinking of at the time, assuming that 1GB RAM alignment to be fairly
> normal. For the !SPARSEMEM_VMEMMAP case, we should probably be fine with
> 29 but, as Will said, we need to be careful with the page flags. At a
> quick look, we have 25 page flags, 2 bits per zone, NUMA nodes and (48 -
> section_size_bits) for the section width. We also need to take into
> account 4 more bits for 52-bit PA support (ARMv8.2). So, without NUMA
> nodes, we are currently at 49 bits used in page->flags.
> 
> For the SPARSEMEM_VMEMMAP case, we can decrease the SECTION_SIZE_BITS in
> the MAX_ORDER limit.
> 
> An alternative would be to free the vmemmap holes later (but still keep
> the vmemmap mapping alias). Yet another option would be to change the
> sparse_mem_map_populate() logic get the actual section end rather than
> always assuming PAGES_PER_SECTION. But I don't think any of these are
> worth if we can safely reduce SECTION_SIZE_BITS.
> 
Yes,
currently,it's safely to reduce the SECTION_SIZE_BITS to match this issue
very well.

As I mentioned before, if the memory layout is not like this scene. There
will be not suitable to reduce the SECTION_SIZE_BITS.

We have 4G memory, and 64GB phys address.

There will be a lot of holes in the memory layout.
And the *holes size are not always the same*.

So,it's the reason I want to enable flat-mem in ARM64-ARCH. Why not makes
the flat-mem an optional setting for arm64i 1/4 ?






--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Catalin Marinas]

Hi Chen,

On Wed, Apr 20, 2016 at 11:18:54AM +0800, Chen Feng wrote:
> Thanks for your reply.
> On 2016/4/12 22:59, Catalin Marinas wrote:
> > On Mon, Apr 11, 2016 at 12:31:53PM +0200, Ard Biesheuvel wrote:
> >> On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
> >>> On 2016/4/11 16:00, Ard Biesheuvel wrote:
> >>>> On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
> >>>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
> >>>>>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
> >>>>>>>  0             1.5G    2G             3.5G            4G
> >>>>>>>  |              |      |               |              |
> >>>>>>>  +--------------+------+---------------+--------------+
> >>>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
> >>>>>>>  +--------------+------+---------------+--------------+
> >>>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
> >>>>
> >>>> No, it is not. It may be covered by a section, but that does not mean
> >>>> sparsemem vmemmap will actually allocate backing for it. The
> >>>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
> >>>> due to the fact that the backing is performed at PMD granularity.
> >>>>
> >>>> Please, could you share the contents of the vmemmap section in
> >>>> /sys/kernel/debug/kernel_page_tables of your system running with
> >>>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
> >>>
> >>> Please see the pg-tables below.
> >>>
> >>> With sparse and vmemmap enable.
> >>>
> >>> ---[ vmemmap start ]---
> >>> 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
> >>> ---[ vmemmap end ]---
> > [...]
> >>> The board is 4GB, and the memap is 70MB
> >>> 1G memory --- 14MB mem_map array.
> >>
> >> No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
> >> pages assuming sizeof(struct page) == 64
> >>
> >> So you are losing 6 MB to rounding here, which I agree is significant.
> >> I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
> >> on 4k pages kernels, but perhaps we're better off asking the opinion
> >> of the other cc'ees.
> > 
> > IIRC, SECTION_SIZE_BITS was chosen to be the maximum sane value we were
> > thinking of at the time, assuming that 1GB RAM alignment to be fairly
> > normal. For the !SPARSEMEM_VMEMMAP case, we should probably be fine with
> > 29 but, as Will said, we need to be careful with the page flags. At a
> > quick look, we have 25 page flags, 2 bits per zone, NUMA nodes and (48 -
> > section_size_bits) for the section width. We also need to take into
> > account 4 more bits for 52-bit PA support (ARMv8.2). So, without NUMA
> > nodes, we are currently at 49 bits used in page->flags.
> > 
> > For the SPARSEMEM_VMEMMAP case, we can decrease the SECTION_SIZE_BITS in
> > the MAX_ORDER limit.
> > 
> > An alternative would be to free the vmemmap holes later (but still keep
> > the vmemmap mapping alias). Yet another option would be to change the
> > sparse_mem_map_populate() logic get the actual section end rather than
> > always assuming PAGES_PER_SECTION. But I don't think any of these are
> > worth if we can safely reduce SECTION_SIZE_BITS.
> 
> Yes,
> currently,it's safely to reduce the SECTION_SIZE_BITS to match this issue
> very well.
> 
> As I mentioned before, if the memory layout is not like this scene. There
> will be not suitable to reduce the SECTION_SIZE_BITS.

SECTION_SIZE_BITS is not meant to cover all possible combinations but
only sane ones and it was primarily targeted at the ARM memory map
recommendations:

http://infocenter.arm.com/help/topic/com.arm.doc.den0001c/DEN0001C_principles_of_arm_memory_maps.pdf

As you know have reported a platform that uses half GB-aligned RAM
blocks/sizes, I'm fine to change SECTION_SIZE_BITS. If in the future we
see even more insane configurations and the memory wasted is
significant, we may have to revisit this (I also proposed an alternative
above like freeing the vmmemap holes, that's not too different from a
flat memmap array).

> We have 4G memory, and 64GB phys address.
> 
> There will be a lot of holes in the memory layout.
> And the *holes size are not always the same*.

It's not the hole size that matters but rather the section size and
alignment.

> So,it's the reason I want to enable flat-mem in ARM64-ARCH. Why not makes
> the flat-mem an optional setting for arm64i 1/4 ?

Because (a) I strongly believe in single Image, (b) I do not want to
increase the configuration space unnecessarily (already large enough
with all the page and VA size combinations) and (c) I don't see any
advantage in flatmem compared to sparsemem+vmemap.

-- 
Catalin

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Chen Feng]

On 2016/4/11 17:59, Chen Feng wrote:
> Hi Ard,
> 
> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>> On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
>>> Hi Ard,
>>>
>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>>> Hi will,
>>>>> Thanks for review.
>>>>>
>>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>>> We can reduce the memory allocated at mem-map
>>>>>>> by flatmem.
>>>>>>>
>>>>>>> currently, the default memory-model in arm64 is
>>>>>>> sparse memory. The mem-map array is not freed in
>>>>>>> this scene. If the physical address is too long,
>>>>>>> it will reserved too much memory for the mem-map
>>>>>>> array.
>>>>>>
>>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>>
>>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>>> is not freed.
>>>>>
>>>>> take a look at here:
>>>>> arm64/mm/init.c
>>>>> void __init mem_init(void)
>>>>> {
>>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>>>         free_unused_memmap();
>>>>> #endif
>>>>> }
>>>>>
>>>>> Memory layout (3GB)
>>>>>
>>>>>  0             1.5G    2G             3.5G            4G
>>>>>  |              |      |               |              |
>>>>>  +--------------+------+---------------+--------------+
>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>  +--------------+------+---------------+--------------+
>>>>>
>>>>>
>>>>> Memory layout (4GB)
>>>>>
>>>>>  0                                    3.5G            4G    4.5G
>>>>>  |                                     |              |       |
>>>>>  +-------------------------------------+--------------+-------+
>>>>>  |                   MEM               |   IO (regs)  |  MEM  |
>>>>>  +-------------------------------------+--------------+-------+
>>>>>
>>>>> Currently, the sparse memory section is 1GB.
>>>>>
>>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>>
>>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>>
>>>>
>>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>>> for memory regions that are actually populated.
>>>>
>>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>>> something like this in the boot log
>>>>
>>>> [    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>>> (     8 GB maximum)
>>>> [    0.000000]               0xffffffbdc0000000 - 0xffffffbde2000000
>>>> (   544 MB actual)
>>>>
>>>> but in reality, only the following regions have been allocated
>>>>
>>>> ---[ vmemmap start ]---
>>>> 0xffffffbdc0000000-0xffffffbdc2000000          32M       RW NX SHD AF
>>>>       BLK UXN MEM/NORMAL
>>>> 0xffffffbde0000000-0xffffffbde2000000          32M       RW NX SHD AF
>>>>       BLK UXN MEM/NORMAL
>>>> ---[ vmemmap end ]---
>>>>
>>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>>> all.
>>>
>>> Yes,but the populate is section, which is 1GB. Take a look at the above
>>> memory layout.
>>>
>>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>>
>>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>>>  0             1.5G    2G             3.5G            4G
>>>>>  |              |      |               |              |
>>>>>  +--------------+------+---------------+--------------+
>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>  +--------------+------+---------------+--------------+
>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>
>>
>> No, it is not. It may be covered by a section, but that does not mean
>> sparsemem vmemmap will actually allocate backing for it. The
>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>> due to the fact that the backing is performed at PMD granularity.
>>
>> Please, could you share the contents of the vmemmap section in
>> /sys/kernel/debug/kernel_page_tables of your system running with
>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>
> 
> Please see the pg-tables below.
> 
> 
> With sparse and vmemmap enable.
> 
> ---[ vmemmap start ]---
> 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
> ---[ vmemmap end ]---
> 
> 
> The board is 4GB, and the memap is 70MB
> 1G memory --- 14MB mem_map array.
> So the 4GB has 5 sections, which used 5 * 14MB memory.
> 
>
Sorry, 1G memory is 16GB
5 sections is 5 * 16 = 80MB

1G / 4K * (struct page) 64B = 16MB

I don't know why the vmemap dump in pg-tables is 70MB.

I add hack code in vmemmap_populate sparse_mem_map_populate.

here is the log:
sparse_mem_map_populate 188 start ffffffbdc0000000 end ffffffbdc1000000 PAGES_PER_SECTION 40000 nid 0
vmemmap_populate 549 size 200000 total 200000 addr ffffffbdc0000000
vmemmap_populate 549 size 200000 total 400000 addr ffffffbdc0200000
vmemmap_populate 549 size 200000 total 600000 addr ffffffbdc0400000
vmemmap_populate 549 size 200000 total 800000 addr ffffffbdc0600000
vmemmap_populate 549 size 200000 total a00000 addr ffffffbdc0800000
vmemmap_populate 549 size 200000 total c00000 addr ffffffbdc0a00000
vmemmap_populate 549 size 200000 total e00000 addr ffffffbdc0c00000
vmemmap_populate 549 size 200000 total 1000000 addr ffffffbdc0e00000
sparse_mem_map_populate 188 start ffffffbdc1000000 end ffffffbdc2000000 PAGES_PER_SECTION 40000 nid 0
...
sparse_mem_map_populate 188 start ffffffbdc2000000 end ffffffbdc3000000 PAGES_PER_SECTION 40000 nid 0
sparse_mem_map_populate 188 start ffffffbdc3000000 end ffffffbdc4000000 PAGES_PER_SECTION 40000 nid 0
sparse_mem_map_populate 188 start ffffffbdc4000000 end ffffffbdc5000000 PAGES_PER_SECTION 40000 nid 0


With 4GB memory, it allocated 2MB *  8  * 5 = 80MB.
>  0                                    3.5G            4G    4.5G
>  |                                     |              |       |
>  +-------------------------------------+--------------+-------+
>  |                   MEM               |   IO (regs)  |  MEM  |
>  +-------------------------------------+--------------+-------+

4GB memory ,5 sections. 80MB mem_map allocated.

> 
> 
> 
> 
>> .
>>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Ard Biesheuvel]

On 11 April 2016 at 12:48, Chen Feng <puck.chen@hisilicon.com> wrote:
>
>
> On 2016/4/11 17:59, Chen Feng wrote:
>> Hi Ard,
>>
>> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>>> On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>> Hi Ard,
>>>>
>>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>>>> Hi will,
>>>>>> Thanks for review.
>>>>>>
>>>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>>>> We can reduce the memory allocated at mem-map
>>>>>>>> by flatmem.
>>>>>>>>
>>>>>>>> currently, the default memory-model in arm64 is
>>>>>>>> sparse memory. The mem-map array is not freed in
>>>>>>>> this scene. If the physical address is too long,
>>>>>>>> it will reserved too much memory for the mem-map
>>>>>>>> array.
>>>>>>>
>>>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>>>
>>>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>>>> is not freed.
>>>>>>
>>>>>> take a look at here:
>>>>>> arm64/mm/init.c
>>>>>> void __init mem_init(void)
>>>>>> {
>>>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>>>>         free_unused_memmap();
>>>>>> #endif
>>>>>> }
>>>>>>
>>>>>> Memory layout (3GB)
>>>>>>
>>>>>>  0             1.5G    2G             3.5G            4G
>>>>>>  |              |      |               |              |
>>>>>>  +--------------+------+---------------+--------------+
>>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>>  +--------------+------+---------------+--------------+
>>>>>>
>>>>>>
>>>>>> Memory layout (4GB)
>>>>>>
>>>>>>  0                                    3.5G            4G    4.5G
>>>>>>  |                                     |              |       |
>>>>>>  +-------------------------------------+--------------+-------+
>>>>>>  |                   MEM               |   IO (regs)  |  MEM  |
>>>>>>  +-------------------------------------+--------------+-------+
>>>>>>
>>>>>> Currently, the sparse memory section is 1GB.
>>>>>>
>>>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>>>
>>>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>>>
>>>>>
>>>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>>>> for memory regions that are actually populated.
>>>>>
>>>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>>>> something like this in the boot log
>>>>>
>>>>> [    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>>>> (     8 GB maximum)
>>>>> [    0.000000]               0xffffffbdc0000000 - 0xffffffbde2000000
>>>>> (   544 MB actual)
>>>>>
>>>>> but in reality, only the following regions have been allocated
>>>>>
>>>>> ---[ vmemmap start ]---
>>>>> 0xffffffbdc0000000-0xffffffbdc2000000          32M       RW NX SHD AF
>>>>>       BLK UXN MEM/NORMAL
>>>>> 0xffffffbde0000000-0xffffffbde2000000          32M       RW NX SHD AF
>>>>>       BLK UXN MEM/NORMAL
>>>>> ---[ vmemmap end ]---
>>>>>
>>>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>>>> all.
>>>>
>>>> Yes,but the populate is section, which is 1GB. Take a look at the above
>>>> memory layout.
>>>>
>>>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>>>
>>>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>>>>  0             1.5G    2G             3.5G            4G
>>>>>>  |              |      |               |              |
>>>>>>  +--------------+------+---------------+--------------+
>>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>>  +--------------+------+---------------+--------------+
>>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>>
>>>
>>> No, it is not. It may be covered by a section, but that does not mean
>>> sparsemem vmemmap will actually allocate backing for it. The
>>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>>> due to the fact that the backing is performed at PMD granularity.
>>>
>>> Please, could you share the contents of the vmemmap section in
>>> /sys/kernel/debug/kernel_page_tables of your system running with
>>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>>
>>
>> Please see the pg-tables below.
>>
>>
>> With sparse and vmemmap enable.
>>
>> ---[ vmemmap start ]---
>> 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
>> ---[ vmemmap end ]---
>>
>>
>> The board is 4GB, and the memap is 70MB
>> 1G memory --- 14MB mem_map array.
>> So the 4GB has 5 sections, which used 5 * 14MB memory.
>>
>>
> Sorry, 1G memory is 16GB
> 5 sections is 5 * 16 = 80MB
>
> 1G / 4K * (struct page) 64B = 16MB
>
> I don't know why the vmemap dump in pg-tables is 70MB.
>

It may be the PTDUMP code that emits the vmemmap start marker
incorrectly. Could you please double check?

> I add hack code in vmemmap_populate sparse_mem_map_populate.
>
> here is the log:
> sparse_mem_map_populate 188 start ffffffbdc0000000 end ffffffbdc1000000 PAGES_PER_SECTION 40000 nid 0
> vmemmap_populate 549 size 200000 total 200000 addr ffffffbdc0000000
> vmemmap_populate 549 size 200000 total 400000 addr ffffffbdc0200000
> vmemmap_populate 549 size 200000 total 600000 addr ffffffbdc0400000
> vmemmap_populate 549 size 200000 total 800000 addr ffffffbdc0600000
> vmemmap_populate 549 size 200000 total a00000 addr ffffffbdc0800000
> vmemmap_populate 549 size 200000 total c00000 addr ffffffbdc0a00000
> vmemmap_populate 549 size 200000 total e00000 addr ffffffbdc0c00000
> vmemmap_populate 549 size 200000 total 1000000 addr ffffffbdc0e00000
> sparse_mem_map_populate 188 start ffffffbdc1000000 end ffffffbdc2000000 PAGES_PER_SECTION 40000 nid 0
> ...
> sparse_mem_map_populate 188 start ffffffbdc2000000 end ffffffbdc3000000 PAGES_PER_SECTION 40000 nid 0
> sparse_mem_map_populate 188 start ffffffbdc3000000 end ffffffbdc4000000 PAGES_PER_SECTION 40000 nid 0
> sparse_mem_map_populate 188 start ffffffbdc4000000 end ffffffbdc5000000 PAGES_PER_SECTION 40000 nid 0
>
>
> With 4GB memory, it allocated 2MB *  8  * 5 = 80MB.
>>  0                                    3.5G            4G    4.5G
>>  |                                     |              |       |
>>  +-------------------------------------+--------------+-------+
>>  |                   MEM               |   IO (regs)  |  MEM  |
>>  +-------------------------------------+--------------+-------+
>
> 4GB memory ,5 sections. 80MB mem_map allocated.
>

I suppose using

#define SECTION_SIZE_BITS      29

in arch/arm64/include/asm/sparsemem.h would get rid of the overhead
completely in this particular case. Could you confirm, please?

@Will: is the rationale for the default value of 30 for
SECTION_SIZE_BITS documented anywhere? Compared to other
architectures, it seems on the high side, but I did notice that 64k
granule kernels require at least 28 in order not to trigger the
following assertion

include/linux/mmzone.h:1029:2: error: #error Allocator MAX_ORDER
exceeds SECTION_SIZE

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

[Jungseok Lee]

On Apr 11, 2016, at 11:49 AM, Chen Feng wrote:

Dear Chen,

> Hi will,
> Thanks for review.
> 
> On 2016/4/7 22:21, Will Deacon wrote:
>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>> We can reduce the memory allocated at mem-map
>>> by flatmem.
>>> 
>>> currently, the default memory-model in arm64 is
>>> sparse memory. The mem-map array is not freed in
>>> this scene. If the physical address is too long,
>>> it will reserved too much memory for the mem-map
>>> array.
>> 
>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>> spaces between memory banks only burns up virtual space. What exactly is
>> the problem you're seeing that makes you want to use flatmem (which is
>> probably unsuitable for the majority of arm64 machines).
>> 
> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
> is not freed.
> 
> take a look at here:
> arm64/mm/init.c
> void __init mem_init(void)
> {
> #ifndef CONFIG_SPARSEMEM_VMEMMAP
> 	free_unused_memmap();
> #endif
> }
> 
> Memory layout (3GB)
> 
> 0             1.5G    2G             3.5G            4G
> |              |      |               |              |
> +--------------+------+---------------+--------------+
> |    MEM       | hole |     MEM       |   IO (regs)  |
> +--------------+------+---------------+--------------+
> 
> 
> Memory layout (4GB)
> 
> 0                                    3.5G            4G    4.5G
> |                                     |              |       |
> +-------------------------------------+--------------+-------+
> |                   MEM               |   IO (regs)  |  MEM  |
> +-------------------------------------+--------------+-------+
> 
> Currently, the sparse memory section is 1GB.
> 
> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
> 
> This will alloc 1G/4K * (struct page) memory for mem_map array.
> 
> We want to use flat-mem to reduce the alloced mem_map.
> 
> I don't know why you tell us the flatmem is unsuitable for the
> majority of arm64 machines. Can tell us the reason of it?
> 
> And we are not going to limit the memdel in arm64, we just want to
> make the flat-mem is an optional item in arm64.

I've experienced the same problem and considered the ideas mentioned
in this thread: flatmem and small SECTION_SIZE_BITS. However, I was
reluctant to post any patch since the issue is highly related to memory
map design document, [1], saying 1GB aligned RAM. The majority of arm64
platforms might follow the information although it is not spec. IOW,
a machine I've played was at least unusual *at that time*, so I didn't
consider upstream work.

[1] http://infocenter.arm.com/help/topic/com.arm.doc.den0001c/DEN0001C_principles_of_arm_memory_maps.pdf  

Best Regards
Jungseok Lee
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>