why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Xishi Qiu]

I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
There are 8 cpus, 2 zones in my system.

I think may be the pagevec trigger the problem, but PAGEVEC_SIZE is only 14.
Does anyone know the reason?

Thanks,
Xishi Qiu

root@hi3650:/ # cat /proc/vmstat 
nr_free_pages 54192
nr_inactive_anon 39830
nr_active_anon 28794
nr_inactive_file 432444
nr_active_file 20659
nr_unevictable 2363
nr_mlock 0
nr_anon_pages 65249
nr_mapped 19742
nr_file_pages 462723
nr_dirty 20
nr_writeback 0
...


nr_inactive_file 432444
nr_active_file 20659
total is 453103

nr_file_pages 462723

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Xishi Qiu]

I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
There are 8 cpus, 2 zones in my system.

I think may be the pagevec trigger the problem, but PAGEVEC_SIZE is only 14.
Does anyone know the reason?

Thanks,
Xishi Qiu

root@hi3650:/ # cat /proc/vmstat 
nr_free_pages 54192
nr_inactive_anon 39830
nr_active_anon 28794
nr_inactive_file 432444
nr_active_file 20659
nr_unevictable 2363
nr_mlock 0
nr_anon_pages 65249
nr_mapped 19742
nr_file_pages 462723
nr_dirty 20
nr_writeback 0
...


nr_inactive_file 432444
nr_active_file 20659
total is 453103

nr_file_pages 462723

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Aaron Lu]

On 05/13/2016 11:58 AM, Xishi Qiu wrote:
> I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
> There are 8 cpus, 2 zones in my system.
> 
> I think may be the pagevec trigger the problem, but PAGEVEC_SIZE is only 14.
> Does anyone know the reason?

One thing I can see is the ram backed filesystem where the page is
counted as NR_FILE_PAGE but go into the anonymous LRU list instead of
the file LRU list.

See function shmem_getpage_gfp.

An example:
[aaron@aaronlu ~]$ head -11 /proc/vmstat 
nr_free_pages 194472
nr_alloc_batch 58
nr_inactive_anon 483386
nr_active_anon 298161
nr_inactive_file 452791
nr_active_file 1942376
nr_unevictable 84
nr_mlock 84
nr_anon_pages 445332
nr_mapped 93553
nr_file_pages 2731481
[aaron@aaronlu ~]$ fallocate -l 400M /dev/shm/test
[aaron@aaronlu ~]$ head -11 /proc/vmstat 
nr_free_pages 94808
nr_alloc_batch 838
nr_inactive_anon 582385
nr_active_anon 298371
nr_inactive_file 452795
nr_active_file 1942380
nr_unevictable 84
nr_mlock 84
nr_anon_pages 445543
nr_mapped 93658
nr_file_pages 2830488

The nr_file_pages increased with nr_inactive_anon while the
nr_{in}active_file don't see much change.

Regards,
Aaron

> 
> Thanks,
> Xishi Qiu
> 
> root@hi3650:/ # cat /proc/vmstat 
> nr_free_pages 54192
> nr_inactive_anon 39830
> nr_active_anon 28794
> nr_inactive_file 432444
> nr_active_file 20659
> nr_unevictable 2363
> nr_mlock 0
> nr_anon_pages 65249
> nr_mapped 19742
> nr_file_pages 462723
> nr_dirty 20
> nr_writeback 0
> ...
> 
> 
> nr_inactive_file 432444
> nr_active_file 20659
> total is 453103
> 
> nr_file_pages 462723
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Aaron Lu]

On 05/13/2016 11:58 AM, Xishi Qiu wrote:
> I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
> There are 8 cpus, 2 zones in my system.
> 
> I think may be the pagevec trigger the problem, but PAGEVEC_SIZE is only 14.
> Does anyone know the reason?

One thing I can see is the ram backed filesystem where the page is
counted as NR_FILE_PAGE but go into the anonymous LRU list instead of
the file LRU list.

See function shmem_getpage_gfp.

An example:
[aaron@aaronlu ~]$ head -11 /proc/vmstat 
nr_free_pages 194472
nr_alloc_batch 58
nr_inactive_anon 483386
nr_active_anon 298161
nr_inactive_file 452791
nr_active_file 1942376
nr_unevictable 84
nr_mlock 84
nr_anon_pages 445332
nr_mapped 93553
nr_file_pages 2731481
[aaron@aaronlu ~]$ fallocate -l 400M /dev/shm/test
[aaron@aaronlu ~]$ head -11 /proc/vmstat 
nr_free_pages 94808
nr_alloc_batch 838
nr_inactive_anon 582385
nr_active_anon 298371
nr_inactive_file 452795
nr_active_file 1942380
nr_unevictable 84
nr_mlock 84
nr_anon_pages 445543
nr_mapped 93658
nr_file_pages 2830488

The nr_file_pages increased with nr_inactive_anon while the
nr_{in}active_file don't see much change.

Regards,
Aaron

> 
> Thanks,
> Xishi Qiu
> 
> root@hi3650:/ # cat /proc/vmstat 
> nr_free_pages 54192
> nr_inactive_anon 39830
> nr_active_anon 28794
> nr_inactive_file 432444
> nr_active_file 20659
> nr_unevictable 2363
> nr_mlock 0
> nr_anon_pages 65249
> nr_mapped 19742
> nr_file_pages 462723
> nr_dirty 20
> nr_writeback 0
> ...
> 
> 
> nr_inactive_file 432444
> nr_active_file 20659
> total is 453103
> 
> nr_file_pages 462723
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Xishi Qiu]

On 2016/5/13 15:00, Aaron Lu wrote:

> On 05/13/2016 11:58 AM, Xishi Qiu wrote:
>> I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
>> There are 8 cpus, 2 zones in my system.
>>
>> I think may be the pagevec trigger the problem, but PAGEVEC_SIZE is only 14.
>> Does anyone know the reason?
> 
> One thing I can see is the ram backed filesystem where the page is
> counted as NR_FILE_PAGE but go into the anonymous LRU list instead of
> the file LRU list.
> 
> See function shmem_getpage_gfp.
> 
> An example:
> [aaron@aaronlu ~]$ head -11 /proc/vmstat 
> nr_free_pages 194472
> nr_alloc_batch 58
> nr_inactive_anon 483386
> nr_active_anon 298161
> nr_inactive_file 452791
> nr_active_file 1942376
> nr_unevictable 84
> nr_mlock 84
> nr_anon_pages 445332
> nr_mapped 93553
> nr_file_pages 2731481
> [aaron@aaronlu ~]$ fallocate -l 400M /dev/shm/test
> [aaron@aaronlu ~]$ head -11 /proc/vmstat 
> nr_free_pages 94808
> nr_alloc_batch 838
> nr_inactive_anon 582385
> nr_active_anon 298371
> nr_inactive_file 452795
> nr_active_file 1942380
> nr_unevictable 84
> nr_mlock 84
> nr_anon_pages 445543
> nr_mapped 93658
> nr_file_pages 2830488
> 
> The nr_file_pages increased with nr_inactive_anon while the
> nr_{in}active_file don't see much change.
> 
> Regards,
> Aaron
> 

Hi Aaron,

Thanks for your reply, but I find the count of nr_shmem is very small
in my system.

root@hi3650:/ # cat /proc/vmstat 
nr_free_pages 54192
nr_inactive_anon 39830
nr_active_anon 28794
nr_inactive_file 432444
nr_active_file 20659
nr_unevictable 2363
nr_mlock 0
nr_anon_pages 65249
nr_mapped 19742
nr_file_pages 462723
nr_dirty 20
nr_writeback 0
nr_slab_reclaimable 259333
nr_slab_unreclaimable 33463
nr_page_table_pages 3456
nr_kernel_stack 892
nr_unstable 0
nr_bounce 11
nr_vmscan_write 292032
nr_vmscan_immediate_reclaim 47204474
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 128
nr_dirtied 69574
nr_written 356299
nr_anon_transparent_hugepages 0
nr_free_cma 7519
nr_swapcache 41972
nr_dirty_threshold 6982
nr_dirty_background_threshold 99297


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Xishi Qiu]

On 2016/5/13 15:00, Aaron Lu wrote:

> On 05/13/2016 11:58 AM, Xishi Qiu wrote:
>> I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
>> There are 8 cpus, 2 zones in my system.
>>
>> I think may be the pagevec trigger the problem, but PAGEVEC_SIZE is only 14.
>> Does anyone know the reason?
> 
> One thing I can see is the ram backed filesystem where the page is
> counted as NR_FILE_PAGE but go into the anonymous LRU list instead of
> the file LRU list.
> 
> See function shmem_getpage_gfp.
> 
> An example:
> [aaron@aaronlu ~]$ head -11 /proc/vmstat 
> nr_free_pages 194472
> nr_alloc_batch 58
> nr_inactive_anon 483386
> nr_active_anon 298161
> nr_inactive_file 452791
> nr_active_file 1942376
> nr_unevictable 84
> nr_mlock 84
> nr_anon_pages 445332
> nr_mapped 93553
> nr_file_pages 2731481
> [aaron@aaronlu ~]$ fallocate -l 400M /dev/shm/test
> [aaron@aaronlu ~]$ head -11 /proc/vmstat 
> nr_free_pages 94808
> nr_alloc_batch 838
> nr_inactive_anon 582385
> nr_active_anon 298371
> nr_inactive_file 452795
> nr_active_file 1942380
> nr_unevictable 84
> nr_mlock 84
> nr_anon_pages 445543
> nr_mapped 93658
> nr_file_pages 2830488
> 
> The nr_file_pages increased with nr_inactive_anon while the
> nr_{in}active_file don't see much change.
> 
> Regards,
> Aaron
> 

Hi Aaron,

Thanks for your reply, but I find the count of nr_shmem is very small
in my system.

root@hi3650:/ # cat /proc/vmstat 
nr_free_pages 54192
nr_inactive_anon 39830
nr_active_anon 28794
nr_inactive_file 432444
nr_active_file 20659
nr_unevictable 2363
nr_mlock 0
nr_anon_pages 65249
nr_mapped 19742
nr_file_pages 462723
nr_dirty 20
nr_writeback 0
nr_slab_reclaimable 259333
nr_slab_unreclaimable 33463
nr_page_table_pages 3456
nr_kernel_stack 892
nr_unstable 0
nr_bounce 11
nr_vmscan_write 292032
nr_vmscan_immediate_reclaim 47204474
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 128
nr_dirtied 69574
nr_written 356299
nr_anon_transparent_hugepages 0
nr_free_cma 7519
nr_swapcache 41972
nr_dirty_threshold 6982
nr_dirty_background_threshold 99297

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Michal Hocko]

[Sorry I haven't noticed this answer before]

On Fri 13-05-16 15:31:42, Xishi Qiu wrote:
> On 2016/5/13 15:00, Aaron Lu wrote:
> 
> Hi Aaron,
> 
> Thanks for your reply, but I find the count of nr_shmem is very small
> in my system.

which kernel version is this? I remember that we used to account thp
pages as NR_FILE_PAGE as well in the past.

I didn't get to look at your number more closely though.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Michal Hocko]

[Sorry I haven't noticed this answer before]

On Fri 13-05-16 15:31:42, Xishi Qiu wrote:
> On 2016/5/13 15:00, Aaron Lu wrote:
> 
> Hi Aaron,
> 
> Thanks for your reply, but I find the count of nr_shmem is very small
> in my system.

which kernel version is this? I remember that we used to account thp
pages as NR_FILE_PAGE as well in the past.

I didn't get to look at your number more closely though.
-- 
Michal Hocko
SUSE Labs

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Xishi Qiu]

On 2016/5/16 17:57, Michal Hocko wrote:

> [Sorry I haven't noticed this answer before]
> 
> On Fri 13-05-16 15:31:42, Xishi Qiu wrote:
>> On 2016/5/13 15:00, Aaron Lu wrote:
>>
>> Hi Aaron,
>>
>> Thanks for your reply, but I find the count of nr_shmem is very small
>> in my system.
> 
> which kernel version is this? I remember that we used to account thp
> pages as NR_FILE_PAGE as well in the past.
> 
> I didn't get to look at your number more closely though.

Hi Michal,

It's android kernel, v3.10
I think the thp config is off.

Thanks,
Xishi Qiu

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Xishi Qiu]

On 2016/5/16 17:57, Michal Hocko wrote:

> [Sorry I haven't noticed this answer before]
> 
> On Fri 13-05-16 15:31:42, Xishi Qiu wrote:
>> On 2016/5/13 15:00, Aaron Lu wrote:
>>
>> Hi Aaron,
>>
>> Thanks for your reply, but I find the count of nr_shmem is very small
>> in my system.
> 
> which kernel version is this? I remember that we used to account thp
> pages as NR_FILE_PAGE as well in the past.
> 
> I didn't get to look at your number more closely though.

Hi Michal,

It's android kernel, v3.10
I think the thp config is off.

Thanks,
Xishi Qiu

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Michal Hocko]

On Mon 16-05-16 18:01:40, Xishi Qiu wrote:
> On 2016/5/16 17:57, Michal Hocko wrote:
> 
> > [Sorry I haven't noticed this answer before]
> > 
> > On Fri 13-05-16 15:31:42, Xishi Qiu wrote:
> >> On 2016/5/13 15:00, Aaron Lu wrote:
> >>
> >> Hi Aaron,
> >>
> >> Thanks for your reply, but I find the count of nr_shmem is very small
> >> in my system.
> > 
> > which kernel version is this? I remember that we used to account thp
> > pages as NR_FILE_PAGE as well in the past.
> > 
> > I didn't get to look at your number more closely though.
> 
> Hi Michal,
> 
> It's android kernel, v3.10
> I think the thp config is off.

Ble. Not enough sleep. I didn't mean thp but hugetlb pages. Sorry about
the confusion. If even that is not the case then there is either an
accounting bug or some fs doesn't put pages in the pagecache directly to
the LRUs or thos pages are on unevictable list.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Michal Hocko]

On Mon 16-05-16 18:01:40, Xishi Qiu wrote:
> On 2016/5/16 17:57, Michal Hocko wrote:
> 
> > [Sorry I haven't noticed this answer before]
> > 
> > On Fri 13-05-16 15:31:42, Xishi Qiu wrote:
> >> On 2016/5/13 15:00, Aaron Lu wrote:
> >>
> >> Hi Aaron,
> >>
> >> Thanks for your reply, but I find the count of nr_shmem is very small
> >> in my system.
> > 
> > which kernel version is this? I remember that we used to account thp
> > pages as NR_FILE_PAGE as well in the past.
> > 
> > I didn't get to look at your number more closely though.
> 
> Hi Michal,
> 
> It's android kernel, v3.10
> I think the thp config is off.

Ble. Not enough sleep. I didn't mean thp but hugetlb pages. Sorry about
the confusion. If even that is not the case then there is either an
accounting bug or some fs doesn't put pages in the pagecache directly to
the LRUs or thos pages are on unevictable list.
-- 
Michal Hocko
SUSE Labs

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Michal Hocko]

On Fri 13-05-16 11:58:16, Xishi Qiu wrote:
> I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
> There are 8 cpus, 2 zones in my system.

Because they count shmem pages as well and those are living on the anon
lru list (see shmem_add_to_page_cache).
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Michal Hocko]

On Fri 13-05-16 11:58:16, Xishi Qiu wrote:
> I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
> There are 8 cpus, 2 zones in my system.

Because they count shmem pages as well and those are living on the anon
lru list (see shmem_add_to_page_cache).
-- 
Michal Hocko
SUSE Labs

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Xishi Qiu]

On 2016/5/16 17:31, Michal Hocko wrote:

> On Fri 13-05-16 11:58:16, Xishi Qiu wrote:
>> I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
>> There are 8 cpus, 2 zones in my system.
> 
> Because they count shmem pages as well and those are living on the anon
> lru list (see shmem_add_to_page_cache).

Hi Michal,

But the shmem seems very small.

nr_inactive_file 432444
nr_active_file 20659
nr_unevictable 2363
nr_shmem 128

nr_file_pages 462723

There is still 7129 pages difference.

root@hi3650:/ # cat /proc/vmstat 
nr_free_pages 54192
nr_inactive_anon 39830
nr_active_anon 28794
nr_inactive_file 432444
nr_active_file 20659
nr_unevictable 2363
nr_mlock 0
nr_anon_pages 65249
nr_mapped 19742
nr_file_pages 462723
nr_dirty 20
nr_writeback 0
nr_slab_reclaimable 259333
nr_slab_unreclaimable 33463
nr_page_table_pages 3456
nr_kernel_stack 892
nr_unstable 0
nr_bounce 11
nr_vmscan_write 292032
nr_vmscan_immediate_reclaim 47204474
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 128
nr_dirtied 69574
nr_written 356299
nr_anon_transparent_hugepages 0
nr_free_cma 7519
nr_swapcache 41972
nr_dirty_threshold 6982
nr_dirty_background_threshold 99297

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why the count nr_file_pages is not equal to nr_inactive_file + nr_active_file ?

[Xishi Qiu]

On 2016/5/16 17:31, Michal Hocko wrote:

> On Fri 13-05-16 11:58:16, Xishi Qiu wrote:
>> I find the count nr_file_pages is not equal to nr_inactive_file + nr_active_file.
>> There are 8 cpus, 2 zones in my system.
> 
> Because they count shmem pages as well and those are living on the anon
> lru list (see shmem_add_to_page_cache).

Hi Michal,

But the shmem seems very small.

nr_inactive_file 432444
nr_active_file 20659
nr_unevictable 2363
nr_shmem 128

nr_file_pages 462723

There is still 7129 pages difference.

root@hi3650:/ # cat /proc/vmstat 
nr_free_pages 54192
nr_inactive_anon 39830
nr_active_anon 28794
nr_inactive_file 432444
nr_active_file 20659
nr_unevictable 2363
nr_mlock 0
nr_anon_pages 65249
nr_mapped 19742
nr_file_pages 462723
nr_dirty 20
nr_writeback 0
nr_slab_reclaimable 259333
nr_slab_unreclaimable 33463
nr_page_table_pages 3456
nr_kernel_stack 892
nr_unstable 0
nr_bounce 11
nr_vmscan_write 292032
nr_vmscan_immediate_reclaim 47204474
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 128
nr_dirtied 69574
nr_written 356299
nr_anon_transparent_hugepages 0
nr_free_cma 7519
nr_swapcache 41972
nr_dirty_threshold 6982
nr_dirty_background_threshold 99297