[PATCH] mm: walk the zone in pageblock_nr_pages steps

[PATCH] mm: walk the zone in pageblock_nr_pages steps

[zhongjiang]

From: zhong jiang <zhongjiang@huawei.com>

when walking the zone, we can happens to the holes. we should not
align MAX_ORDER_NR_PAGES, so it can skip the normal memory.

In addition, pagetypeinfo_showmixedcount_print reflect fragmentization.
we hope to get more accurate data. therefore, I decide to fix it.

Signed-off-by: zhong jiang <zhongjiang@huawei.com>
---
 mm/vmstat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/vmstat.c b/mm/vmstat.c
index cb2a67b..3508f74 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1033,7 +1033,7 @@ static void pagetypeinfo_showmixedcount_print(struct seq_file *m,
 	 */
 	for (; pfn < end_pfn; ) {
 		if (!pfn_valid(pfn)) {
-			pfn = ALIGN(pfn + 1, MAX_ORDER_NR_PAGES);
+			pfn = ALIGN(pfn + 1, pageblock_nr_pages);
 			continue;
 		}
 
-- 
1.8.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] mm: walk the zone in pageblock_nr_pages steps

[zhongjiang]

From: zhong jiang <zhongjiang@huawei.com>

when walking the zone, we can happens to the holes. we should not
align MAX_ORDER_NR_PAGES, so it can skip the normal memory.

In addition, pagetypeinfo_showmixedcount_print reflect fragmentization.
we hope to get more accurate data. therefore, I decide to fix it.

Signed-off-by: zhong jiang <zhongjiang@huawei.com>
---
 mm/vmstat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/vmstat.c b/mm/vmstat.c
index cb2a67b..3508f74 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1033,7 +1033,7 @@ static void pagetypeinfo_showmixedcount_print(struct seq_file *m,
 	 */
 	for (; pfn < end_pfn; ) {
 		if (!pfn_valid(pfn)) {
-			pfn = ALIGN(pfn + 1, MAX_ORDER_NR_PAGES);
+			pfn = ALIGN(pfn + 1, pageblock_nr_pages);
 			continue;
 		}
 
-- 
1.8.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: walk the zone in pageblock_nr_pages steps

[Vlastimil Babka]

On 07/26/2016 05:08 AM, zhongjiang wrote:
> From: zhong jiang <zhongjiang@huawei.com>
>
> when walking the zone, we can happens to the holes. we should not
> align MAX_ORDER_NR_PAGES, so it can skip the normal memory.
>
> In addition, pagetypeinfo_showmixedcount_print reflect fragmentization.
> we hope to get more accurate data. therefore, I decide to fix it.

Can't say I'm happy with another random half-fix. What's the real 
granularity of holes for CONFIG_HOLES_IN_ZONE systems? I suspect it can 
be below pageblock_nr_pages. The pfn_valid_within() mechanism seems 
rather insufficient... it does prevent running unexpectedly into holes 
in the middle of pageblock/MAX_ORDER block, but together with the large 
skipping it doesn't guarantee that we cover all non-holes.

I think in a robust solution, functions such as these should use 
something like PAGE_HOLE_GRANULARITY which equals MAX_ORDER_NR_PAGES for 
!CONFIG_HOLES_IN_ZONE and some arch/config/system specific value for 
CONFIG_HOLES_IN_ZONE. This would then be used in the ALIGN() part.
It could be also used together with pfn_valid_within() in the inner loop 
to skip over holes more quickly (if it's worth).

Also I just learned there's also CONFIG_ARCH_HAS_HOLES_MEMORYMODEL that 
affects a function called memmap_valid_within(). But that one has only 
one caller - pagetypeinfo_showblockcount_print(). Why is it needed there 
and not in pagetypeinfo_showmixedcount_print() (or anywhere else?)

> Signed-off-by: zhong jiang <zhongjiang@huawei.com>
> ---
>  mm/vmstat.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index cb2a67b..3508f74 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1033,7 +1033,7 @@ static void pagetypeinfo_showmixedcount_print(struct seq_file *m,
>  	 */
>  	for (; pfn < end_pfn; ) {
>  		if (!pfn_valid(pfn)) {
> -			pfn = ALIGN(pfn + 1, MAX_ORDER_NR_PAGES);
> +			pfn = ALIGN(pfn + 1, pageblock_nr_pages);
>  			continue;
>  		}
>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Re: [PATCH] mm: walk the zone in pageblock_nr_pages steps

[zhong jiang]

On 2016/7/26 14:24, Vlastimil Babka wrote:
> On 07/26/2016 05:08 AM, zhongjiang wrote:
>> From: zhong jiang <zhongjiang@huawei.com>
>>
>> when walking the zone, we can happens to the holes. we should not
>> align MAX_ORDER_NR_PAGES, so it can skip the normal memory.
>>
>> In addition, pagetypeinfo_showmixedcount_print reflect fragmentization.
>> we hope to get more accurate data. therefore, I decide to fix it.
>
> Can't say I'm happy with another random half-fix. What's the real granularity of holes for CONFIG_HOLES_IN_ZONE systems? I suspect it can be below pageblock_nr_pages. The pfn_valid_within() mechanism seems rather insufficient... it does prevent running unexpectedly into holes in the middle of pageblock/MAX_ORDER block, but together with the large skipping it doesn't guarantee that we cover all non-holes.
>
  I am sorry for that. I did not review the whole code before sending above patch.  In arch of x86, The real granularity of holes is in 256, that is a section. while in arm64, we can see that the hole is identify by located in
  SYSTEM_RAM. I admit that that is not a best way. but at present, it's a better way to amend.
> I think in a robust solution, functions such as these should use something like PAGE_HOLE_GRANULARITY which equals MAX_ORDER_NR_PAGES for !CONFIG_HOLES_IN_ZONE and some arch/config/system specific value for CONFIG_HOLES_IN_ZONE. This would then be used in the ALIGN() part.
> It could be also used together with pfn_valid_within() in the inner loop to skip over holes more quickly (if it's worth).
>
 Maybe reimplement the code about hole punch is a better way.
> Also I just learned there's also CONFIG_ARCH_HAS_HOLES_MEMORYMODEL that affects a function called memmap_valid_within(). But that one has only one caller - pagetypeinfo_showblockcount_print(). Why is it needed there and not in pagetypeinfo_showmixedcount_print() (or anywhere else?)
>
 yes, but in other place, for example, the caller apagetypeinfo_showmixedcount_print you can see the commit.(91c43c7313a995a8908f8f6b911a85d00fdbffd)
>> Signed-off-by: zhong jiang <zhongjiang@huawei.com>
>> ---
>>  mm/vmstat.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/mm/vmstat.c b/mm/vmstat.c
>> index cb2a67b..3508f74 100644
>> --- a/mm/vmstat.c
>> +++ b/mm/vmstat.c
>> @@ -1033,7 +1033,7 @@ static void pagetypeinfo_showmixedcount_print(struct seq_file *m,
>>       */
>>      for (; pfn < end_pfn; ) {
>>          if (!pfn_valid(pfn)) {
>> -            pfn = ALIGN(pfn + 1, MAX_ORDER_NR_PAGES);
>> +            pfn = ALIGN(pfn + 1, pageblock_nr_pages);
>>              continue;
>>          }
>>
>>
>
> -- 
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: walk the zone in pageblock_nr_pages steps

[Vlastimil Babka]

On 07/26/2016 10:31 AM, zhong jiang wrote:
> On 2016/7/26 14:24, Vlastimil Babka wrote:
>> On 07/26/2016 05:08 AM, zhongjiang wrote:
>>> From: zhong jiang <zhongjiang@huawei.com>
>>>
>>> when walking the zone, we can happens to the holes. we should
>>> not align MAX_ORDER_NR_PAGES, so it can skip the normal memory.
>>>
>>> In addition, pagetypeinfo_showmixedcount_print reflect
>>> fragmentization. we hope to get more accurate data. therefore, I
>>> decide to fix it.
>>
>> Can't say I'm happy with another random half-fix. What's the real
>> granularity of holes for CONFIG_HOLES_IN_ZONE systems? I suspect it
>> can be below pageblock_nr_pages. The pfn_valid_within() mechanism
>> seems rather insufficient... it does prevent running unexpectedly
>> into holes in the middle of pageblock/MAX_ORDER block, but together
>> with the large skipping it doesn't guarantee that we cover all
>> non-holes.
>>
> I am sorry for that. I did not review the whole code before sending
> above patch.  In arch of x86, The real granularity of holes is in
> 256, that is a section.

Huh, x86 doesn't even have CONFIG_HOLES_IN_ZONE? So any pfn valid within 
MAX_ORDER_NR_PAGES (and within zone boundaries?) should mean the whole 
range is valid? AFAICS only ia64, mips and s390 has CONFIG_HOLES_IN_ZONE.

Maybe I misunderstand... can you help by demonstrating on which arch and 
configuration your patch makes a difference?

> while in arm64, we can see that the hole is
> identify by located in SYSTEM_RAM. I admit that that is not a best
> way. but at present, it's a better way to amend.
>> I think in a robust solution, functions such as these should use
>> something like PAGE_HOLE_GRANULARITY which equals
>> MAX_ORDER_NR_PAGES for !CONFIG_HOLES_IN_ZONE and some
>> arch/config/system specific value for CONFIG_HOLES_IN_ZONE. This
>> would then be used in the ALIGN() part. It could be also used
>> together with pfn_valid_within() in the inner loop to skip over
>> holes more quickly (if it's worth).
>>
> Maybe reimplement the code about hole punch is a better way.
>> Also I just learned there's also CONFIG_ARCH_HAS_HOLES_MEMORYMODEL
>> that affects a function called memmap_valid_within(). But that one
>> has only one caller - pagetypeinfo_showblockcount_print(). Why is
>> it needed there and not in pagetypeinfo_showmixedcount_print() (or
>> anywhere else?)
>>
> yes, but in other place, for example, the caller
> apagetypeinfo_showmixedcount_print you can see the
> commit.(91c43c7313a995a8908f8f6b911a85d00fdbffd)

Hmm I don't see such commit in linus.git, mmotm or linux-next trees.

>>> Signed-off-by: zhong jiang <zhongjiang@huawei.com> ---
>>> mm/vmstat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/mm/vmstat.c b/mm/vmstat.c index cb2a67b..3508f74
>>> 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1033,7 +1033,7 @@
>>> static void pagetypeinfo_showmixedcount_print(struct seq_file
>>> *m, */ for (; pfn < end_pfn; ) { if (!pfn_valid(pfn)) { -
>>> pfn = ALIGN(pfn + 1, MAX_ORDER_NR_PAGES); +            pfn =
>>> ALIGN(pfn + 1, pageblock_nr_pages); continue; }
>>>
>>>
>>
>> -- To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@kvack.org.  For more info on Linux MM, see:
>> http://www.linux-mm.org/ . Don't email: <a
>> href=mailto:"dont@kvack.org"> email@kvack.org </a>
>>
>>
>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: walk the zone in pageblock_nr_pages steps

[zhong jiang]

On 2016/7/26 16:53, Vlastimil Babka wrote:
> On 07/26/2016 10:31 AM, zhong jiang wrote:
>> On 2016/7/26 14:24, Vlastimil Babka wrote:
>>> On 07/26/2016 05:08 AM, zhongjiang wrote:
>>>> From: zhong jiang <zhongjiang@huawei.com>
>>>>
>>>> when walking the zone, we can happens to the holes. we should
>>>> not align MAX_ORDER_NR_PAGES, so it can skip the normal memory.
>>>>
>>>> In addition, pagetypeinfo_showmixedcount_print reflect
>>>> fragmentization. we hope to get more accurate data. therefore, I
>>>> decide to fix it.
>>>
>>> Can't say I'm happy with another random half-fix. What's the real
>>> granularity of holes for CONFIG_HOLES_IN_ZONE systems? I suspect it
>>> can be below pageblock_nr_pages. The pfn_valid_within() mechanism
>>> seems rather insufficient... it does prevent running unexpectedly
>>> into holes in the middle of pageblock/MAX_ORDER block, but together
>>> with the large skipping it doesn't guarantee that we cover all
>>> non-holes.
>>>
>> I am sorry for that. I did not review the whole code before sending
>> above patch.  In arch of x86, The real granularity of holes is in
>> 256, that is a section.
>
> Huh, x86 doesn't even have CONFIG_HOLES_IN_ZONE? So any pfn valid within MAX_ORDER_NR_PAGES (and within zone boundaries?) should mean the whole range is valid? AFAICS only ia64, mips and s390 has CONFIG_HOLES_IN_ZONE.
>
> Maybe I misunderstand... can you help by demonstrating on which arch and configuration your patch makes a difference?
>
 a x86 arch ,for example, when CONFIG_HOLES_IN_ZONE disable, hole punch is not existence. we scan the zone in the way of pageblock ,compared with the MAX_ORDER_NR_PAGES, it should be more resonable.
 when CONFIG_HOLES_IN_ZONE enable, hole punch is existence. it will prevent the rest 2M to be skipped. you can get from code that we prefer to align with pageblock.
>> while in arm64, we can see that the hole is
>> identify by located in SYSTEM_RAM. I admit that that is not a best
>> way. but at present, it's a better way to amend.
>>> I think in a robust solution, functions such as these should use
>>> something like PAGE_HOLE_GRANULARITY which equals
>>> MAX_ORDER_NR_PAGES for !CONFIG_HOLES_IN_ZONE and some
>>> arch/config/system specific value for CONFIG_HOLES_IN_ZONE. This
>>> would then be used in the ALIGN() part. It could be also used
>>> together with pfn_valid_within() in the inner loop to skip over
>>> holes more quickly (if it's worth).
>>>
>> Maybe reimplement the code about hole punch is a better way.
>>> Also I just learned there's also CONFIG_ARCH_HAS_HOLES_MEMORYMODEL
>>> that affects a function called memmap_valid_within(). But that one
>>> has only one caller - pagetypeinfo_showblockcount_print(). Why is
>>> it needed there and not in pagetypeinfo_showmixedcount_print() (or
>>> anywhere else?)
>>>
>> yes, but in other place, for example, the caller
>> apagetypeinfo_showmixedcount_print you can see the
>> commit.(91c43c7313a995a8908f8f6b911a85d00fdbffd)
>
> Hmm I don't see such commit in linus.git, mmotm or linux-next trees.
>
>>>> Signed-off-by: zhong jiang <zhongjiang@huawei.com> ---
>>>> mm/vmstat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/mm/vmstat.c b/mm/vmstat.c index cb2a67b..3508f74
>>>> 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1033,7 +1033,7 @@
>>>> static void pagetypeinfo_showmixedcount_print(struct seq_file
>>>> *m, */ for (; pfn < end_pfn; ) { if (!pfn_valid(pfn)) { -
>>>> pfn = ALIGN(pfn + 1, MAX_ORDER_NR_PAGES); +            pfn =
>>>> ALIGN(pfn + 1, pageblock_nr_pages); continue; }
>>>>
>>>>
>>>
>>> -- To unsubscribe, send a message with 'unsubscribe linux-mm' in
>>> the body to majordomo@kvack.org.  For more info on Linux MM, see:
>>> http://www.linux-mm.org/ . Don't email: <a
>>> href=mailto:"dont@kvack.org"> email@kvack.org </a>
>>>
>>>
>>
>>
>
>
> .
>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: walk the zone in pageblock_nr_pages steps

[Joonsoo Kim]

On Tue, Jul 26, 2016 at 05:30:59PM +0800, zhong jiang wrote:
> On 2016/7/26 16:53, Vlastimil Babka wrote:
> > On 07/26/2016 10:31 AM, zhong jiang wrote:
> >> On 2016/7/26 14:24, Vlastimil Babka wrote:
> >>> On 07/26/2016 05:08 AM, zhongjiang wrote:
> >>>> From: zhong jiang <zhongjiang@huawei.com>
> >>>>
> >>>> when walking the zone, we can happens to the holes. we should
> >>>> not align MAX_ORDER_NR_PAGES, so it can skip the normal memory.

Do you have any system to trigger this problem?

I'm not familiar with CONFIG_HOLES_IN_ZONE system, but, as Vlastimil saids,
skip by pageblock size also has similar problem that skip the normal memory
because hole's granularity would not be pageblock size.

Anyway, if you want not to skip the normal memory, following code would work.
I think that it is a better way since it doesn't depend on hole's granularity.

Thanks.

--------->8-----------
diff --git a/mm/vmstat.c b/mm/vmstat.c
index e1a4690..4184db2 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1276,6 +1276,11 @@ static void pagetypeinfo_showmixedcount_print(struct seq_file *m,
         * not matter as the mixed block count will still be correct
         */
        for (; pfn < end_pfn; ) {
+               if (!pfn_valid_within(pfn)) {
+                       pfn++;
+                       continue;
+               }
+
                if (!pfn_valid(pfn)) {
                        pfn = ALIGN(pfn + 1, MAX_ORDER_NR_PAGES);
                        continue;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: walk the zone in pageblock_nr_pages steps

[zhong jiang]

On 2016/7/28 14:57, Joonsoo Kim wrote:
> On Tue, Jul 26, 2016 at 05:30:59PM +0800, zhong jiang wrote:
>> On 2016/7/26 16:53, Vlastimil Babka wrote:
>>> On 07/26/2016 10:31 AM, zhong jiang wrote:
>>>> On 2016/7/26 14:24, Vlastimil Babka wrote:
>>>>> On 07/26/2016 05:08 AM, zhongjiang wrote:
>>>>>> From: zhong jiang <zhongjiang@huawei.com>
>>>>>>
>>>>>> when walking the zone, we can happens to the holes. we should
>>>>>> not align MAX_ORDER_NR_PAGES, so it can skip the normal memory.
> Do you have any system to trigger this problem?
>
> I'm not familiar with CONFIG_HOLES_IN_ZONE system, but, as Vlastimil saids,
> skip by pageblock size also has similar problem that skip the normal memory
> because hole's granularity would not be pageblock size.
>
> Anyway, if you want not to skip the normal memory, following code would work.
> I think that it is a better way since it doesn't depend on hole's granularity.
>
> Thanks.
  you maybe get me wrong. page type is showed with block size as the unit.  it is not skip the
  normal memory but we should align block. the following code will waste of time to skip the
  hole.  because it maybe need to realign to pageblock.
  I just want to express that align with pageblock will consistence with the following code. 
  and it maybe lead to skip the normal memory.
 
  Thanks
> --------->8-----------
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index e1a4690..4184db2 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1276,6 +1276,11 @@ static void pagetypeinfo_showmixedcount_print(struct seq_file *m,
>          * not matter as the mixed block count will still be correct
>          */
>         for (; pfn < end_pfn; ) {
> +               if (!pfn_valid_within(pfn)) {
> +                       pfn++;
> +                       continue;
> +               }
> +
>                 if (!pfn_valid(pfn)) {
>                         pfn = ALIGN(pfn + 1, MAX_ORDER_NR_PAGES);
>                         continue;
>
>
> .
>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>