[PATCH V2] powerpc/Kconfig: Update config option based on page size.

[PATCH V2] powerpc/Kconfig: Update config option based on page size.

[Rashmica Gupta]

Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.

The error occurs because of the following constraint (from
include/linux/mmzone.h) being violated:

	MAX_ORDER -1 + PAGESHIFT <= SECTION_SIZE_BITS.

Expanding this out, we get:

	FORCE_MAX_ZONEBITS <= 25 - PAGESHIFT,

which requires, for a 64K page, FORCE_MAX_ZONEBITS <= 9. Thus set max
value of FORCE_MAX_ZONEORDER for 64K pages to 9, and 4K pages to 13.

Also, check the minimum value:
In include/linux/huge_mm.h, we have the constraint HPAGE_PMD_ORDER <
MAX_ORDER which expands out to:

	PTE_INDEX_SIZE < FORCE_MAX_ZONEORDER.

PTE_INDEX_SIZE is:
	9 (4k hash or no hash 4K pgtable) or
	8 (64K hash or no hash 64K pgtable).
Thus a min value of 8 for 64K pages and 9 for 4K pages is reasonable.

So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
and from 13-64 to 9-13 for 4K pages.

Signed-off-by: Rashmica Gupta <rashmicy@gmail.com>
---

v2: Changed the range for 4K pages and minimum for 64K pages as suggested
by Balbir Singh. 


 arch/powerpc/Kconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index e4824fd04bb7..b933530821fb 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -585,9 +585,9 @@ endchoice
 
 config FORCE_MAX_ZONEORDER
 	int "Maximum zone order"
-	range 9 64 if PPC64 && PPC_64K_PAGES
+	range 8 9 if PPC64 && PPC_64K_PAGES
 	default "9" if PPC64 && PPC_64K_PAGES
-	range 13 64 if PPC64 && !PPC_64K_PAGES
+	range 9 13 if PPC64 && !PPC_64K_PAGES
 	default "13" if PPC64 && !PPC_64K_PAGES
 	range 9 64 if PPC32 && PPC_16K_PAGES
 	default "9" if PPC32 && PPC_16K_PAGES
-- 
2.5.0

Re: [PATCH V2] powerpc/Kconfig: Update config option based on page size.

[Balbir Singh]

On 19/02/16 16:38, Rashmica Gupta wrote:
> Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
> FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.
>
> The error occurs because of the following constraint (from
> include/linux/mmzone.h) being violated:
>
> 	MAX_ORDER -1 + PAGESHIFT <= SECTION_SIZE_BITS.
>
> Expanding this out, we get:
>
> 	FORCE_MAX_ZONEBITS <= 25 - PAGESHIFT,
>
> which requires, for a 64K page, FORCE_MAX_ZONEBITS <= 9. Thus set max
> value of FORCE_MAX_ZONEORDER for 64K pages to 9, and 4K pages to 13.
>
> Also, check the minimum value:
> In include/linux/huge_mm.h, we have the constraint HPAGE_PMD_ORDER <
> MAX_ORDER which expands out to:
>
> 	PTE_INDEX_SIZE < FORCE_MAX_ZONEORDER.
>
> PTE_INDEX_SIZE is:
> 	9 (4k hash or no hash 4K pgtable) or
> 	8 (64K hash or no hash 64K pgtable).
> Thus a min value of 8 for 64K pages and 9 for 4K pages is reasonable.
>
> So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
> and from 13-64 to 9-13 for 4K pages.
>
> Signed-off-by: Rashmica Gupta <rashmicy@gmail.com>
> ---
>
> v2: Changed the range for 4K pages and minimum for 64K pages as suggested
> by Balbir Singh. 
>
>
>  arch/powerpc/Kconfig | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index e4824fd04bb7..b933530821fb 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -585,9 +585,9 @@ endchoice
>  
>  config FORCE_MAX_ZONEORDER
>  	int "Maximum zone order"
> -	range 9 64 if PPC64 && PPC_64K_PAGES
> +	range 8 9 if PPC64 && PPC_64K_PAGES
>  	default "9" if PPC64 && PPC_64K_PAGES
> -	range 13 64 if PPC64 && !PPC_64K_PAGES
> +	range 9 13 if PPC64 && !PPC_64K_PAGES
>  	default "13" if PPC64 && !PPC_64K_PAGES
>  	range 9 64 if PPC32 && PPC_16K_PAGES
>  	default "9" if PPC32 && PPC_16K_PAGES
Reviewed-by: Balbir Singh <bsingharora@gmail.com>

Balbir Singh

Re: [V2] powerpc/Kconfig: Update config option based on page size.

[Michael Ellerman]

On Fri, 2016-19-02 at 05:38:47 UTC, Rashmica Gupta wrote:
> Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
> FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.
> 
...
> So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
> and from 13-64 to 9-13 for 4K pages.
> 
> Signed-off-by: Rashmica Gupta <rashmicy@gmail.com>
> Reviewed-by: Balbir Singh <bsingharora@gmail.com>

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/a7ee539584acf4a565b7439cea

cheers

Re: [V2] powerpc/Kconfig: Update config option based on page size.

[Aneesh Kumar K.V]

Michael Ellerman <mpe@ellerman.id.au> writes:

> On Fri, 2016-19-02 at 05:38:47 UTC, Rashmica Gupta wrote:
>> Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
>> FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.
>> 
> ...
>> So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
>> and from 13-64 to 9-13 for 4K pages.
>> 
>> Signed-off-by: Rashmica Gupta <rashmicy@gmail.com>
>> Reviewed-by: Balbir Singh <bsingharora@gmail.com>
>
> Applied to powerpc next, thanks.
>
> https://git.kernel.org/powerpc/c/a7ee539584acf4a565b7439cea
>

HPAGE_PMD_ORDER is not something we should check w.r.t 4k linux page
size. We do have the below constraint w.r.t hugetlb pages

static inline bool hstate_is_gigantic(struct hstate *h)
{
	return huge_page_order(h) >= MAX_ORDER;
}

That require MAX_ORDER to be greater than 12.

Did we test hugetlbfs 4k config with this patch ? Will it work if we
start marking hugepage as gigantic page ?

-aneesh

Re: [V2] powerpc/Kconfig: Update config option based on page size.

[Balbir Singh]

On 20/04/16 00:59, Aneesh Kumar K.V wrote:
> Michael Ellerman <mpe@ellerman.id.au> writes:
> 
>> On Fri, 2016-19-02 at 05:38:47 UTC, Rashmica Gupta wrote:
>>> Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
>>> FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.
>>>
>> ...
>>> So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
>>> and from 13-64 to 9-13 for 4K pages.
>>>
>>> Signed-off-by: Rashmica Gupta <rashmicy@gmail.com>
>>> Reviewed-by: Balbir Singh <bsingharora@gmail.com>
>>
>> Applied to powerpc next, thanks.
>>
>> https://git.kernel.org/powerpc/c/a7ee539584acf4a565b7439cea
>>
> 
> HPAGE_PMD_ORDER is not something we should check w.r.t 4k linux page
> size. We do have the below constraint w.r.t hugetlb pages
> 
> static inline bool hstate_is_gigantic(struct hstate *h)
> {
> 	return huge_page_order(h) >= MAX_ORDER;
> }
> 
> That require MAX_ORDER to be greater than 12.
> 

The build will fail for MAX_ZONEORDER beyond the specified limits.
MAX_ORDER > 12 for what page size?

My understanding is this

1. gigantic refers to the fact the regular allocators cannot allocate
this page
2. Use alloc_contig_range() with CONFIG_CMA for gigantic pages

I could be wrong

> Did we test hugetlbfs 4k config with this patch ? Will it work if we
> start marking hugepage as gigantic page ?

Nope.. I did not

Thanks for the review!
Balbir Singh

Re: [V2] powerpc/Kconfig: Update config option based on page size.

[santhosh]

[-- Attachment #1: Type: text/plain, Size: 2068 bytes --]


> Michael Ellerman <mpe@ellerman.id.au> writes:
>
>> On Fri, 2016-19-02 at 05:38:47 UTC, Rashmica Gupta wrote:
>>> Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
>>> FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.
>>>
>> ...
>>> So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
>>> and from 13-64 to 9-13 for 4K pages.
>>>
>>> Signed-off-by: Rashmica Gupta <rashmicy@gmail.com>
>>> Reviewed-by: Balbir Singh <bsingharora@gmail.com>
>> Applied to powerpc next, thanks.
>>
>> https://git.kernel.org/powerpc/c/a7ee539584acf4a565b7439cea
>>
> HPAGE_PMD_ORDER is not something we should check w.r.t 4k linux page
> size. We do have the below constraint w.r.t hugetlb pages
>
> static inline bool hstate_is_gigantic(struct hstate *h)
> {
> 	return huge_page_order(h) >= MAX_ORDER;
> }
>
> That require MAX_ORDER to be greater than 12.
>
> Did we test hugetlbfs 4k config with this patch ? Will it work if we
> start marking hugepage as gigantic page ?
>
> -aneesh
>
Hello Rashmica,

With upstream linux kernel 4.8.0-rc1-00006-gbae9cc6 compiled with linux 
4k page size we are not able set hugepages, Aneesh had a look at the 
problem and he mentioned this commit is causing the issue.

*Details:*
We are using pkvm ubuntu 16.04 guest with upstream kernel 
[4.8.0-rc1-00006-gbae9cc6] compiled with  4k page size

o/p from guest:
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:      16384 kB

Page sizes from device-tree: [dmesg]
[    0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, 
tlbiel=1, penc=0
[    0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, 
tlbiel=1, penc=56
[    0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, 
tlbiel=0, penc=0

while trying to configure the hugepages inside the guest it throws the 
below error:

echo 100 > /proc/sys/vm/nr_hugepages
-bash: echo: write error: Invalid argument

*Note*: we do not see the problem when the linux page is 64k

Thanks,
Santhosh G


[-- Attachment #2: Type: text/html, Size: 3279 bytes --]

Re: [V2] powerpc/Kconfig: Update config option based on page size.

[Michael Ellerman]

santhosh <santhog4@linux.vnet.ibm.com> writes:

>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>
>>> On Fri, 2016-19-02 at 05:38:47 UTC, Rashmica Gupta wrote:
>>>> Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
>>>> FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.
>>>>
>>> ...
>>>> So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
>>>> and from 13-64 to 9-13 for 4K pages.
>>>
>>> https://git.kernel.org/powerpc/c/a7ee539584acf4a565b7439cea
>>>
>> HPAGE_PMD_ORDER is not something we should check w.r.t 4k linux page
>> size. We do have the below constraint w.r.t hugetlb pages
>>
>> static inline bool hstate_is_gigantic(struct hstate *h)
>> {
>> 	return huge_page_order(h) >= MAX_ORDER;
>> }
>>
>> That require MAX_ORDER to be greater than 12.

So have you tried that fix?

cheers

Re: [V2] powerpc/Kconfig: Update config option based on page size.

[Balbir Singh]

On 14/09/16 20:40, santhosh wrote:
> 
>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>
>>> On Fri, 2016-19-02 at 05:38:47 UTC, Rashmica Gupta wrote:
>>>> Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
>>>> FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.
>>>>
>>> ...
>>>> So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
>>>> and from 13-64 to 9-13 for 4K pages.
>>>>
>>>> Signed-off-by: Rashmica Gupta <rashmicy@gmail.com>
>>>> Reviewed-by: Balbir Singh <bsingharora@gmail.com>
>>> Applied to powerpc next, thanks.
>>>
>>> https://git.kernel.org/powerpc/c/a7ee539584acf4a565b7439cea
>>>
>> HPAGE_PMD_ORDER is not something we should check w.r.t 4k linux page
>> size. We do have the below constraint w.r.t hugetlb pages
>>
>> static inline bool hstate_is_gigantic(struct hstate *h)
>> {
>>     return huge_page_order(h) >= MAX_ORDER;
>> }
>>
>> That require MAX_ORDER to be greater than 12.
>>

9 to 13 was done based on calculations you can find the commit



>> Did we test hugetlbfs 4k config with this patch ? Will it work if we
>> start marking hugepage as gigantic page ?
>>
>> -aneesh
>>
> Hello Rashmica,
> 
> With upstream linux kernel 4.8.0-rc1-00006-gbae9cc6 compiled with linux 4k page size we are not able set hugepages, Aneesh had a look at the problem and he mentioned this commit is causing the issue.
> 
> *Details:*
> We are using pkvm ubuntu 16.04 guest with upstream kernel [4.8.0-rc1-00006-gbae9cc6] compiled with  4k page size
> 
> o/p from guest:
> HugePages_Total:       0
> HugePages_Free:        0
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:      16384 kB
> 
> Page sizes from device-tree: [dmesg]
> [    0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
> [    0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
> [    0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0
> 
> while trying to configure the hugepages inside the guest it throws the below error:
> 
> echo 100 > /proc/sys/vm/nr_hugepages
> -bash: echo: write error: Invalid argument
> 
> *Note*: we do not see the problem when the linux page is 64k


Just to reiterate you are seeing this problem using 4k page size and 16M as the hugepage size.
With FORCE_MAX_ZONEORDER set to 9 to 13 for 4k pages, you can do upto 32M if FORCE_MAX_ZONEORDER
is 13 and same for 64K with FORCE_MAX_ZONEORDER set to 9.

Basically the constraint is


FORCE_MAX_ZONEBITS <= 25 - PAGESHIFT

What is your value of FORCE_MAX_ZONEORDER in the .config?

Balbir Singh.

Re: [V2] powerpc/Kconfig: Update config option based on page size.

[Aneesh Kumar K.V]

Balbir Singh <bsingharora@gmail.com> writes:

> On 14/09/16 20:40, santhosh wrote:
>> 
>>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>>
>>>> On Fri, 2016-19-02 at 05:38:47 UTC, Rashmica Gupta wrote:
>>>>> Currently on PPC64 changing kernel pagesize from 4K to 64K leaves
>>>>> FORCE_MAX_ZONEORDER set to 13 - which produces a compile error.
>>>>>
>>>> ...
>>>>> So, update the range of FORCE_MAX_ZONEORDER from 9-64 to 8-9 for 64K pages
>>>>> and from 13-64 to 9-13 for 4K pages.
>>>>>
>>>>> Signed-off-by: Rashmica Gupta <rashmicy@gmail.com>
>>>>> Reviewed-by: Balbir Singh <bsingharora@gmail.com>
>>>> Applied to powerpc next, thanks.
>>>>
>>>> https://git.kernel.org/powerpc/c/a7ee539584acf4a565b7439cea
>>>>
>>> HPAGE_PMD_ORDER is not something we should check w.r.t 4k linux page
>>> size. We do have the below constraint w.r.t hugetlb pages
>>>
>>> static inline bool hstate_is_gigantic(struct hstate *h)
>>> {
>>>     return huge_page_order(h) >= MAX_ORDER;
>>> }
>>>
>>> That require MAX_ORDER to be greater than 12.
>>>
>
> 9 to 13 was done based on calculations you can find the commit
>
>
>
>>> Did we test hugetlbfs 4k config with this patch ? Will it work if we
>>> start marking hugepage as gigantic page ?
>>>
>>> -aneesh
>>>
>> Hello Rashmica,
>> 
>> With upstream linux kernel 4.8.0-rc1-00006-gbae9cc6 compiled with linux 4k page size we are not able set hugepages, Aneesh had a look at the problem and he mentioned this commit is causing the issue.
>> 
>> *Details:*
>> We are using pkvm ubuntu 16.04 guest with upstream kernel [4.8.0-rc1-00006-gbae9cc6] compiled with  4k page size
>> 
>> o/p from guest:
>> HugePages_Total:       0
>> HugePages_Free:        0
>> HugePages_Rsvd:        0
>> HugePages_Surp:        0
>> Hugepagesize:      16384 kB
>> 
>> Page sizes from device-tree: [dmesg]
>> [    0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
>> [    0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
>> [    0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0
>> 
>> while trying to configure the hugepages inside the guest it throws the below error:
>> 
>> echo 100 > /proc/sys/vm/nr_hugepages
>> -bash: echo: write error: Invalid argument
>> 
>> *Note*: we do not see the problem when the linux page is 64k
>
>
> Just to reiterate you are seeing this problem using 4k page size and 16M as the hugepage size.
> With FORCE_MAX_ZONEORDER set to 9 to 13 for 4k pages, you can do upto 32M if FORCE_MAX_ZONEORDER
> is 13 and same for 64K with FORCE_MAX_ZONEORDER set to 9.
>
> Basically the constraint is
>
>
> FORCE_MAX_ZONEBITS <= 25 - PAGESHIFT
>
> What is your value of FORCE_MAX_ZONEORDER in the .config?

The problem is the reverse of why the orginal fix was done. ie, When you
change the pae size from 64K to 4K using nconfig, we don't update the
FORCE_MAX_ZONEORDER and hence we end up a value of 9. That results in
the above error with hugetlb. So from a failed build when switching from
4k to 64K we now have a broken hugetlb when switching from 64K to 4K.

As suggested in the review of the original patch, we should make the 
FORCE_MAX_ZONEORDER range such that it picks the right value that will
get 16MB hugetlb to work.

Something like the below. That range is strange, but without that it picks a value
of 11.

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 927d2ab2ce08..792cb1768c8f 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -637,7 +637,7 @@ config FORCE_MAX_ZONEORDER
 	int "Maximum zone order"
 	range 8 9 if PPC64 && PPC_64K_PAGES
 	default "9" if PPC64 && PPC_64K_PAGES
-	range 9 13 if PPC64 && !PPC_64K_PAGES
+	range 13 13 if PPC64 && !PPC_64K_PAGES
 	default "13" if PPC64 && !PPC_64K_PAGES
 	range 9 64 if PPC32 && PPC_16K_PAGES
 	default "9" if PPC32 && PPC_16K_PAGES


-aneesh

Re: [V2] powerpc/Kconfig: Update config option based on page size.

[Balbir Singh]

On 17/09/16 01:16, Aneesh Kumar K.V wrote:
> -	range 9 13 if PPC64 && !PPC_64K_PAGES
> +	range 13 13 if PPC64 && !PPC_64K_PAGES
>  	default "13" if PPC64 && !PPC_64K_PAGES

Do we still want t to be 13 13 or 12 13? 
Looks like the lower side of the range can cause issues based
on the values of Huge page size


Balbir Singh.