[PATCH 0/1] arm64: mm: add config options for page table configuration

[PATCH 0/1] arm64: mm: add config options for page table configuration

[Scott Branden]

This patch introduces config option CONFIG_MAX_PHYSMEM_BITS and
CONFIG_SECTIONS_SIZE_BITS to allow for memory savings for devices with
small memory areas in sparse memory configurations.

Scott Branden (1):
  arm64: mm: add config options for page table configuration

 arch/arm64/Kconfig                 | 21 +++++++++++++++++++++
 arch/arm64/include/asm/sparsemem.h |  4 ++--
 2 files changed, 23 insertions(+), 2 deletions(-)

-- 
2.5.0

[PATCH 1/1] arm64: mm: add config options for page table configuration

[Scott Branden]

Make MAX_PHYSMEM_BITS and SECTIONS_SIZE_BITS configurable by adding
config options.
Default to current settings currently defined in sparesmem.h.
For systems wishing to save memory the config options can be overridden.
Example, changing MAX_PHYSMEM_BITS from 48 to 36 at the same time as
changing SECTION_SIZE_BITS from 30 to 26 frees 13MB of memory.

Signed-off-by: Scott Branden <scott.branden@broadcom.com>
---
 arch/arm64/Kconfig                 | 21 +++++++++++++++++++++
 arch/arm64/include/asm/sparsemem.h |  4 ++--
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 2482fdd..d0a95de 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -121,6 +121,27 @@ config ARCH_PHYS_ADDR_T_64BIT
 config MMU
 	def_bool y
 
+config MAX_PHYSMEM_BITS
+	int "Maximum Physical Addressable Bits"
+	depends on SPARSEMEM
+	default 48
+	help
+	  Maximum number of physical addressable bits.
+	  If not all the physical bits are used this number can be reduced
+	  to save memory when creating page tables.
+
+	  If unsure, leave this at the default.
+
+config SECTION_SIZE_BITS
+	int "Section Size Bits"
+	depends on SPARSEMEM
+	default 30
+	help
+	  Size of each memory section.
+	  If you need to be able to allocate smaller
+	  sections in page table this number can be change to save memory.
+
+	  If unsure, leave this at the default.
 config DEBUG_RODATA
 	def_bool y
 
diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
index 74a9d30..6677582 100644
--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -17,8 +17,8 @@
 #define __ASM_SPARSEMEM_H
 
 #ifdef CONFIG_SPARSEMEM
-#define MAX_PHYSMEM_BITS	48
-#define SECTION_SIZE_BITS	30
+#define MAX_PHYSMEM_BITS	CONFIG_MAX_PHYSMEM_BITS
+#define SECTION_SIZE_BITS	CONFIG_SECTION_SIZE_BITS
 #endif
 
 #endif
-- 
2.5.0

[PATCH 1/1] arm64: mm: add config options for page table configuration

[Catalin Marinas]

On Wed, Dec 07, 2016 at 11:40:00AM -0800, Scott Branden wrote:
> Make MAX_PHYSMEM_BITS and SECTIONS_SIZE_BITS configurable by adding
> config options.
> Default to current settings currently defined in sparesmem.h.
> For systems wishing to save memory the config options can be overridden.
> Example, changing MAX_PHYSMEM_BITS from 48 to 36 at the same time as
> changing SECTION_SIZE_BITS from 30 to 26 frees 13MB of memory.

I'm not keen on such change, it's a big departure from the single Image
aims. I would rather reduce SECTION_SIZE_BITS permanently where
feasible, like in this patch:

http://lkml.kernel.org/r/1465821119-3384-1-git-send-email-jszhang at marvell.com

-- 
Catalin

[PATCH 1/1] arm64: mm: add config options for page table configuration

[Scott Branden]

Hi Catalin,

On 16-12-08 02:00 AM, Catalin Marinas wrote:
> On Wed, Dec 07, 2016 at 11:40:00AM -0800, Scott Branden wrote:
>> Make MAX_PHYSMEM_BITS and SECTIONS_SIZE_BITS configurable by adding
>> config options.
>> Default to current settings currently defined in sparesmem.h.
>> For systems wishing to save memory the config options can be overridden.
>> Example, changing MAX_PHYSMEM_BITS from 48 to 36 at the same time as
>> changing SECTION_SIZE_BITS from 30 to 26 frees 13MB of memory.
>
> I'm not keen on such change, it's a big departure from the single Image
> aims.
A single Image is not entirely possible when the system needs to be 
tuned for memory usage, boot time, and other performance related issues. 
  These are key features in embedded systems vs. general purpose computers.

I would rather reduce SECTION_SIZE_BITS permanently where
> feasible, like in this patch:
>
> http://lkml.kernel.org/r/1465821119-3384-1-git-send-email-jszhang at marvell.com
>
This patch does not meet my requirements as I need SECTION_SIZE_BITS to 
be set to 28 to reduce memory and to allow memory hotplug to allocate a 
256 MB section.  My patch future proofs the tuning of the parameters by 
allowing any section size to be made.  I could combine the patch you 
list such that SECTION_SIZE_BITS defaults to 30 when 
CONFIG_ARM64_64_PAGES is selected and 27 otherwise.  Should it default 
to something else for 16K and 4K pages?

In terms of MAX_PHYSMEM_BITS, if our SoCs only use 40 (or less) bits I 
would also like the configuration functionality.  This allows us to make 
the SECTION_SIZE_BITS smaller.

Regards,
Scott

[PATCH 1/1] arm64: mm: add config options for page table configuration

[Catalin Marinas]

On Thu, Dec 08, 2016 at 08:30:36AM -0800, Scott Branden wrote:
> On 16-12-08 02:00 AM, Catalin Marinas wrote:
> >On Wed, Dec 07, 2016 at 11:40:00AM -0800, Scott Branden wrote:
> >>Make MAX_PHYSMEM_BITS and SECTIONS_SIZE_BITS configurable by adding
> >>config options.
> >>Default to current settings currently defined in sparesmem.h.
> >>For systems wishing to save memory the config options can be overridden.
> >>Example, changing MAX_PHYSMEM_BITS from 48 to 36 at the same time as
> >>changing SECTION_SIZE_BITS from 30 to 26 frees 13MB of memory.
[...]
> > I would rather reduce SECTION_SIZE_BITS permanently where
> >feasible, like in this patch:
> >
> >http://lkml.kernel.org/r/1465821119-3384-1-git-send-email-jszhang at marvell.com
>
> This patch does not meet my requirements as I need SECTION_SIZE_BITS to be
> set to 28 to reduce memory

So with this patch, we reduce it to 27, it should be fine-grained enough
for 128MB sections. Alternatively, there were other suggestions here:

http://lkml.iu.edu/hypermail/linux/kernel/1604.1/03036.html

> and to allow memory hotplug to allocate a 256 MB section.

Can memory hotplug not work with 2*128MB sections in this case?

> My patch future proofs the tuning of the parameters by allowing
> any section size to be made. 

While MAX_PHYSMEM_BITS makes sense to users in general,
SECTION_SIZE_BITS is not always clear to the average user what it means
and its min/max boundaries. That's another reason (apart from single/few
Image case) why I prefer to not expose it as configuration option.

> I could combine the patch you list such that
> SECTION_SIZE_BITS defaults to 30 when CONFIG_ARM64_64_PAGES is selected and
> 27 otherwise.  Should it default to something else for 16K and 4K pages?

I haven't done any calculations for 16K yet but we could probably come
up with some formula based on PAGE_SHIFT to cover all cases.

> In terms of MAX_PHYSMEM_BITS, if our SoCs only use 40 (or less) bits I would
> also like the configuration functionality.  This allows us to make the
> SECTION_SIZE_BITS smaller.

So how small do you want SECTION_SIZE_BITS to be? As I said above, 128MB
sections should be sufficient in most cases and without the need to
reduce MAX_PHYSMEM_BITS.

-- 
Catalin

[PATCH 1/1] arm64: mm: add config options for page table configuration

[Scott Branden]

On 16-12-08 10:57 AM, Catalin Marinas wrote:
> On Thu, Dec 08, 2016 at 08:30:36AM -0800, Scott Branden wrote:
>> On 16-12-08 02:00 AM, Catalin Marinas wrote:
>>> On Wed, Dec 07, 2016 at 11:40:00AM -0800, Scott Branden wrote:
>>>> Make MAX_PHYSMEM_BITS and SECTIONS_SIZE_BITS configurable by adding
>>>> config options.
>>>> Default to current settings currently defined in sparesmem.h.
>>>> For systems wishing to save memory the config options can be overridden.
>>>> Example, changing MAX_PHYSMEM_BITS from 48 to 36 at the same time as
>>>> changing SECTION_SIZE_BITS from 30 to 26 frees 13MB of memory.
> [...]
>>> I would rather reduce SECTION_SIZE_BITS permanently where
>>> feasible, like in this patch:
>>>
>>> http://lkml.kernel.org/r/1465821119-3384-1-git-send-email-jszhang at marvell.com
>>
>> This patch does not meet my requirements as I need SECTION_SIZE_BITS to be
>> set to 28 to reduce memory
>
> So with this patch, we reduce it to 27, it should be fine-grained enough
> for 128MB sections. Alternatively, there were other suggestions here:
>
> http://lkml.iu.edu/hypermail/linux/kernel/1604.1/03036.html
>
>> and to allow memory hotplug to allocate a 256 MB section.
>
> Can memory hotplug not work with 2*128MB sections in this case?
Yes, I then need to hotplug the memory at 2 locations for 1 memory 
addition but that will work in my current use case.  I'm one step away 
from hotplug working on ARM64.  Once that works I hope to break the 
dependencies between hotplug memory size created based on 
SECTION_SIZE_BITS in the future.

Since I currently have your attention:  I do think there is fundamental 
bug in the ARM64 mm implementation.  If you look at 
/sys/devices/system/memory it only shows the last memoryX section 
created after init.  It should be showing up multiple sections.  As a 
quick test change SECTION_SIZE_BITS and you look at 
/sys/devices/system/memory to see what changes.  Look at a standard x64 
machine you will see all the memoryX entries present.

>
>> My patch future proofs the tuning of the parameters by allowing
>> any section size to be made.
>
> While MAX_PHYSMEM_BITS makes sense to users in general,
> SECTION_SIZE_BITS is not always clear to the average user what it means
> and its min/max boundaries. That's another reason (apart from single/few
> Image case) why I prefer to not expose it as configuration option.
I agree SECTION_SIZE_BITS is confusing.  If you could provide more 
documentation on what it means and how it is used that would help others 
for sure.  I just stumbled upon it while working on a tight memory 
system and found it saves me significant memory.
>
>> I could combine the patch you list such that
>> SECTION_SIZE_BITS defaults to 30 when CONFIG_ARM64_64_PAGES is selected and
>> 27 otherwise.  Should it default to something else for 16K and 4K pages?
>
> I haven't done any calculations for 16K yet but we could probably come
> up with some formula based on PAGE_SHIFT to cover all cases.
Yes, a calculation based on pages that modified SECTION_SIZE_BITS would 
probably be a better solution.
>
>> In terms of MAX_PHYSMEM_BITS, if our SoCs only use 40 (or less) bits I would
>> also like the configuration functionality.  This allows us to make the
>> SECTION_SIZE_BITS smaller.
>
> So how small do you want SECTION_SIZE_BITS to be? As I said above, 128MB
> sections should be sufficient in most cases and without the need to
> reduce MAX_PHYSMEM_BITS.
>
256MB works for my current use case.  But appears somebody else was 
looking for 64MB previously.  So that is why adding support for 
modifying MAX_PHYSMEM_BITS makes sense as it needs to be modified to 
support the 64MB case:
https://lkml.org/lkml/2016/8/11/209

[PATCH 1/1] arm64: mm: add config options for page table configuration

[Will Deacon]

On Thu, Dec 08, 2016 at 11:33:39AM -0800, Scott Branden wrote:
> Since I currently have your attention:  I do think there is fundamental bug
> in the ARM64 mm implementation.  If you look at /sys/devices/system/memory
> it only shows the last memoryX section created after init.

That directory doesn't seem to exist on my arm64 systems. Do I have to
enable something specific in the .config?

Will

[PATCH 1/1] arm64: mm: add config options for page table configuration

[Scott Branden]

Hi Will,

On 16-12-09 02:57 AM, Will Deacon wrote:
> On Thu, Dec 08, 2016 at 11:33:39AM -0800, Scott Branden wrote:
>> Since I currently have your attention:  I do think there is fundamental bug
>> in the ARM64 mm implementation.  If you look at /sys/devices/system/memory
>> it only shows the last memoryX section created after init.
>
> That directory doesn't seem to exist on my arm64 systems. Do I have to
> enable something specific in the .config?
I looked in the /sys/devices/system/memory at it doesn't look like it 
appears until memory hotplug is enabled in the system.  This is another 
patch I'm trying to work through at the same time:
https://www.mail-archive.com/linux-kernel at vger.kernel.org/msg1284943.html

The internals of the memory management subsystem is not something I'm 
too familiar with at this point.
>
> Will
>
Regards,
Scott