Inactive memory keep growing and how to release it?

Inactive memory keep growing and how to release it?

[Lenky Gao]

Hi,

When i just run a test on Centos 6.2 as follows:

#!/bin/bash

while true
do

	file="/tmp/filetest"

	echo $file

	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null

	sleep 5
done

the inactive memory keep growing:

#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   420144 kB
...
#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   911912 kB
...
#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1547484 kB
...

and i cannot reclaim it:

# cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1557684 kB
# echo 3 > /proc/sys/vm/drop_caches
# cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1520832 kB

I have tested on other version kernel, such as 2.6.30 and .6.11, the
problom also exists.

When in the final situation, i cannot kmalloc a larger contiguous
memory, especially in interrupt context.
Can you give some tips to avoid this?

PS:
# uname -a
Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux



--
Regards,

Lenky

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Zlatko Calusic]

On 04.03.2013 10:52, Lenky Gao wrote:
> Hi,
>
> When i just run a test on Centos 6.2 as follows:
>
> #!/bin/bash
>
> while true
> do
>
> 	file="/tmp/filetest"
>
> 	echo $file
>
> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
>
> 	sleep 5
> done
>
> the inactive memory keep growing:
>
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   420144 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   911912 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1547484 kB
> ...
>
> and i cannot reclaim it:
>
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1557684 kB
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1520832 kB
>
> I have tested on other version kernel, such as 2.6.30 and .6.11, the
> problom also exists.
>
> When in the final situation, i cannot kmalloc a larger contiguous
> memory, especially in interrupt context.
> Can you give some tips to avoid this?
>

The drop_caches mechanism doesn't free dirty page cache pages. And your 
bash script is creating a lot of dirty pages. Run it like this and see 
if it helps your case:

sync; echo 3 > /proc/sys/vm/drop_caches

Regards,
-- 
Zlatko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Lenky Gao]

2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>
> The drop_caches mechanism doesn't free dirty page cache pages. And your bash
> script is creating a lot of dirty pages. Run it like this and see if it
> helps your case:
>
> sync; echo 3 > /proc/sys/vm/drop_caches

Thanks for your advice.

The inactive memory still cannot be reclaimed after i execute the sync command:

# cat /proc/meminfo | grep Inactive\(file\);
Inactive(file):   882824 kB
# sync;
# echo 3 > /proc/sys/vm/drop_caches
# cat /proc/meminfo | grep Inactive\(file\);
Inactive(file):   777664 kB

I find these page becomes orphaned in this function, but do not understand why:

/*
 * If truncate cannot remove the fs-private metadata from the page, the page
 * becomes orphaned.  It will be left on the LRU and may even be mapped into
 * user pagetables if we're racing with filemap_fault().
 *
 * We need to bale out if page->mapping is no longer equal to the original
 * mapping.  This happens a) when the VM reclaimed the page while we waited on
 * its lock, b) when a concurrent invalidate_mapping_pages got there first and
 * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
 */
static int
truncate_complete_page(struct address_space *mapping, struct page *page)
{
...

My file system type is ext3, mounted with the opteion data=journal and
it is easy to reproduce.


-- 
Regards,

Lenky

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Will Huck]

Cc experts. Hugh, Johannes,

On 03/04/2013 08:21 PM, Lenky Gao wrote:
> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>> The drop_caches mechanism doesn't free dirty page cache pages. And your bash
>> script is creating a lot of dirty pages. Run it like this and see if it
>> helps your case:
>>
>> sync; echo 3 > /proc/sys/vm/drop_caches
> Thanks for your advice.
>
> The inactive memory still cannot be reclaimed after i execute the sync command:
>
> # cat /proc/meminfo | grep Inactive\(file\);
> Inactive(file):   882824 kB
> # sync;
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(file\);
> Inactive(file):   777664 kB
>
> I find these page becomes orphaned in this function, but do not understand why:
>
> /*
>   * If truncate cannot remove the fs-private metadata from the page, the page
>   * becomes orphaned.  It will be left on the LRU and may even be mapped into
>   * user pagetables if we're racing with filemap_fault().
>   *
>   * We need to bale out if page->mapping is no longer equal to the original
>   * mapping.  This happens a) when the VM reclaimed the page while we waited on
>   * its lock, b) when a concurrent invalidate_mapping_pages got there first and
>   * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>   */
> static int
> truncate_complete_page(struct address_space *mapping, struct page *page)
> {
> ...
>
> My file system type is ext3, mounted with the opteion data=journal and
> it is easy to reproduce.
>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Michal Hocko]

On Mon 04-03-13 17:52:22, Lenky Gao wrote:
> Hi,
> 
> When i just run a test on Centos 6.2 as follows:
> 
> #!/bin/bash
> 
> while true
> do
> 
> 	file="/tmp/filetest"
> 
> 	echo $file
> 
> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
> 
> 	sleep 5
> done
> 
> the inactive memory keep growing:
> 
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   420144 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   911912 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1547484 kB
> ...
> 
> and i cannot reclaim it:

How did you try to reclaim the memory? How much memory is still free?
Are you above watermaks (/proc/zoneinfo will tell you more)

> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1557684 kB
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1520832 kB
> 
> I have tested on other version kernel, such as 2.6.30 and .6.11, the
> problom also exists.
> 
> When in the final situation, i cannot kmalloc a larger contiguous
> memory, especially in interrupt context.

This could be related to the memory fragmentation and your kernel seem
to be too large to have memory compaction which helps a lot in that
area.

> Can you give some tips to avoid this?

One way would be to increase /proc/sys/vm/min_free_kbytes which will
enlarge watermaks so the reclaim starts sooner.
 
> PS:
> # uname -a
> Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
> 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux

This is really an old kernel and also a distribution one which might
contain a lot of patches on top of the core kernel. I would suggest to
contact Redhat or try to reproduce the issue with the vanilla and
up-to-date kernel and report here.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Hillf Danton]

On Sat, Mar 9, 2013 at 10:14 AM, Will Huck <will.huckk@gmail.com> wrote:
> Cc experts. Hugh, Johannes,
>
> On 03/04/2013 08:21 PM, Lenky Gao wrote:
>>
>> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>>>
>>> The drop_caches mechanism doesn't free dirty page cache pages. And your
>>> bash
>>> script is creating a lot of dirty pages. Run it like this and see if it
>>> helps your case:
>>>
>>> sync; echo 3 > /proc/sys/vm/drop_caches
>>
>> Thanks for your advice.
>>
>> The inactive memory still cannot be reclaimed after i execute the sync
>> command:
>>
>> # cat /proc/meminfo | grep Inactive\(file\);
>> Inactive(file):   882824 kB
>> # sync;
>> # echo 3 > /proc/sys/vm/drop_caches
>> # cat /proc/meminfo | grep Inactive\(file\);
>> Inactive(file):   777664 kB
>>
>> I find these page becomes orphaned in this function, but do not understand
>> why:
>>
>> /*
>>   * If truncate cannot remove the fs-private metadata from the page, the
>> page
>>   * becomes orphaned.  It will be left on the LRU and may even be mapped
>> into
>>   * user pagetables if we're racing with filemap_fault().
>>   *
>>   * We need to bale out if page->mapping is no longer equal to the
>> original
>>   * mapping.  This happens a) when the VM reclaimed the page while we
>> waited on
>>   * its lock, b) when a concurrent invalidate_mapping_pages got there
>> first and
>>   * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>>   */
>> static int
>> truncate_complete_page(struct address_space *mapping, struct page *page)
>> {
>> ...
>>
>> My file system type is ext3, mounted with the opteion data=journal and
>> it is easy to reproduce.
>>

Perhaps we have to consider page count for orphan page if it
could be reproduced with mainline.

Hillf
---
--- a/mm/vmscan.c	Sun Mar 10 13:36:26 2013
+++ b/mm/vmscan.c	Thu Mar 14 20:29:40 2013
@@ -315,14 +315,14 @@ out:
 	return ret;
 }

-static inline int is_page_cache_freeable(struct page *page)
+static inline int is_page_cache_freeable(struct page *page, int has_mapping)
 {
 	/*
 	 * A freeable page cache page is referenced only by the caller
 	 * that isolated the page, the page cache radix tree and
 	 * optional buffer heads at page->private.
 	 */
-	return page_count(page) - page_has_private(page) == 2;
+	return page_count(page) - page_has_private(page) == has_mapping + 1;
 }

 static int may_write_to_queue(struct backing_dev_info *bdi,
@@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
 	 * swap_backing_dev_info is bust: it doesn't reflect the
 	 * congestion state of the swapdevs.  Easy to fix, if needed.
 	 */
-	if (!is_page_cache_freeable(page))
+	if (!is_page_cache_freeable(page, mapping ? 1 : 0))
 		return PAGE_KEEP;
 	if (!mapping) {
 		/*
--

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Lenky Gao]

On Thu, Mar 14, 2013 at 6:14 PM, Michal Hocko <mhocko@suse.cz> wrote:
> One way would be to increase /proc/sys/vm/min_free_kbytes which will
> enlarge watermaks so the reclaim starts sooner.
>

Good tip thanks. :)

> This is really an old kernel and also a distribution one which might
> contain a lot of patches on top of the core kernel. I would suggest to
> contact Redhat or try to reproduce the issue with the vanilla and
> up-to-date kernel and report here.

I have tested on other version vanilla kernel, such as 2.6.30 and 3.6.11, the
issue also exist and it is easy to reproduce.

Maybe i have found the answer for this question:

On Thu, Mar 14, 2013 at 4:00 PM, Lenky Gao <lenky.gao@gmail.com> wrote:
> Hi Everyone,
>
> Maybe i have found the answer for this question. The author of the JBD
> have explained in the comments:
>
> /*
>  * When an ext3-ordered file is truncated, it is possible that many pages are
>  * not successfully freed, because they are attached to a committing
> transaction.
>  * After the transaction commits, these pages are left on the LRU, with no
>  * ->mapping, and with attached buffers.  These pages are trivially reclaimable
>  * by the VM, but their apparent absence upsets the VM accounting, and it makes
>  * the numbers in /proc/meminfo look odd.
> ...
>  */
> static void release_buffer_page(struct buffer_head *bh)
> {
>         struct page *page;
> ...

But my new question is why not free those pages directly after the
transaction commits?

On Thu, Mar 14, 2013 at 8:39 PM, Hillf Danton <dhillf@gmail.com> wrote:
> Perhaps we have to consider page count for orphan page if it
> could be reproduced with mainline.
>
> Hillf
> ---
> --- a/mm/vmscan.c       Sun Mar 10 13:36:26 2013
> +++ b/mm/vmscan.c       Thu Mar 14 20:29:40 2013
> @@ -315,14 +315,14 @@ out:
>         return ret;
>  }
>
> -static inline int is_page_cache_freeable(struct page *page)
> +static inline int is_page_cache_freeable(struct page *page, int has_mapping)
>  {
>         /*
>          * A freeable page cache page is referenced only by the caller
>          * that isolated the page, the page cache radix tree and
>          * optional buffer heads at page->private.
>          */
> -       return page_count(page) - page_has_private(page) == 2;
> +       return page_count(page) - page_has_private(page) == has_mapping + 1;
>  }
>
>  static int may_write_to_queue(struct backing_dev_info *bdi,
> @@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
>          * swap_backing_dev_info is bust: it doesn't reflect the
>          * congestion state of the swapdevs.  Easy to fix, if needed.
>          */
> -       if (!is_page_cache_freeable(page))
> +       if (!is_page_cache_freeable(page, mapping ? 1 : 0))
>                 return PAGE_KEEP;
>         if (!mapping) {
>                 /*

Thanks, i'll test it.

I am totally a newbie regarding VMM and EXT/JBD, thanks to everyone
for your kind attention and help.

-- 
Regards,

Lenky

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Simon Jeons]

On 03/14/2013 06:14 PM, Michal Hocko wrote:
> On Mon 04-03-13 17:52:22, Lenky Gao wrote:
>> Hi,
>>
>> When i just run a test on Centos 6.2 as follows:
>>
>> #!/bin/bash
>>
>> while true
>> do
>>
>> 	file="/tmp/filetest"
>>
>> 	echo $file
>>
>> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
>>
>> 	sleep 5
>> done
>>
>> the inactive memory keep growing:
>>
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):   420144 kB
>> ...
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):   911912 kB
>> ...
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1547484 kB
>> ...
>>
>> and i cannot reclaim it:
> How did you try to reclaim the memory? How much memory is still free?
> Are you above watermaks (/proc/zoneinfo will tell you more)
>
>> # cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1557684 kB
>> # echo 3 > /proc/sys/vm/drop_caches
>> # cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1520832 kB
>>
>> I have tested on other version kernel, such as 2.6.30 and .6.11, the
>> problom also exists.
>>
>> When in the final situation, i cannot kmalloc a larger contiguous
>> memory, especially in interrupt context.
> This could be related to the memory fragmentation and your kernel seem
> to be too large to have memory compaction which helps a lot in that
> area.
>
>> Can you give some tips to avoid this?
> One way would be to increase /proc/sys/vm/min_free_kbytes which will
> enlarge watermaks so the reclaim starts sooner.
>   
>> PS:
>> # uname -a
>> Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
>> 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
> This is really an old kernel and also a distribution one which might
> contain a lot of patches on top of the core kernel. I would suggest to
> contact Redhat or try to reproduce the issue with the vanilla and

What's the meaning of vanilla?

> up-to-date kernel and report here.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Simon Jeons]

On 03/14/2013 08:39 PM, Hillf Danton wrote:
> On Sat, Mar 9, 2013 at 10:14 AM, Will Huck <will.huckk@gmail.com> wrote:
>> Cc experts. Hugh, Johannes,
>>
>> On 03/04/2013 08:21 PM, Lenky Gao wrote:
>>> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>>>> The drop_caches mechanism doesn't free dirty page cache pages. And your
>>>> bash
>>>> script is creating a lot of dirty pages. Run it like this and see if it
>>>> helps your case:
>>>>
>>>> sync; echo 3 > /proc/sys/vm/drop_caches
>>> Thanks for your advice.
>>>
>>> The inactive memory still cannot be reclaimed after i execute the sync
>>> command:
>>>
>>> # cat /proc/meminfo | grep Inactive\(file\);
>>> Inactive(file):   882824 kB
>>> # sync;
>>> # echo 3 > /proc/sys/vm/drop_caches
>>> # cat /proc/meminfo | grep Inactive\(file\);
>>> Inactive(file):   777664 kB
>>>
>>> I find these page becomes orphaned in this function, but do not understand
>>> why:
>>>
>>> /*
>>>    * If truncate cannot remove the fs-private metadata from the page, the
>>> page
>>>    * becomes orphaned.  It will be left on the LRU and may even be mapped
>>> into
>>>    * user pagetables if we're racing with filemap_fault().
>>>    *
>>>    * We need to bale out if page->mapping is no longer equal to the
>>> original
>>>    * mapping.  This happens a) when the VM reclaimed the page while we
>>> waited on
>>>    * its lock, b) when a concurrent invalidate_mapping_pages got there
>>> first and
>>>    * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>>>    */
>>> static int
>>> truncate_complete_page(struct address_space *mapping, struct page *page)
>>> {
>>> ...
>>>
>>> My file system type is ext3, mounted with the opteion data=journal and
>>> it is easy to reproduce.
>>>
> Perhaps we have to consider page count for orphan page if it
> could be reproduced with mainline.

Why? /proc/sys/vm/drop_caches will call invalidate_mapping_pages() 
instead of truncate_complete_page().

>
> Hillf
> ---
> --- a/mm/vmscan.c	Sun Mar 10 13:36:26 2013
> +++ b/mm/vmscan.c	Thu Mar 14 20:29:40 2013
> @@ -315,14 +315,14 @@ out:
>   	return ret;
>   }
>
> -static inline int is_page_cache_freeable(struct page *page)
> +static inline int is_page_cache_freeable(struct page *page, int has_mapping)
>   {
>   	/*
>   	 * A freeable page cache page is referenced only by the caller
>   	 * that isolated the page, the page cache radix tree and
>   	 * optional buffer heads at page->private.
>   	 */
> -	return page_count(page) - page_has_private(page) == 2;
> +	return page_count(page) - page_has_private(page) == has_mapping + 1;
>   }

page count 2 is for page cache and isolator, why you check mapping 
separately?

>   static int may_write_to_queue(struct backing_dev_info *bdi,
> @@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
>   	 * swap_backing_dev_info is bust: it doesn't reflect the
>   	 * congestion state of the swapdevs.  Easy to fix, if needed.
>   	 */
> -	if (!is_page_cache_freeable(page))
> +	if (!is_page_cache_freeable(page, mapping ? 1 : 0))
>   		return PAGE_KEEP;
>   	if (!mapping) {
>   		/*
> --
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Theodore Ts'o]

On Fri, Mar 15, 2013 at 04:41:41PM +0800, Simon Jeons wrote:
> >This is really an old kernel and also a distribution one which might
> >contain a lot of patches on top of the core kernel. I would suggest to
> >contact Redhat or try to reproduce the issue with the vanilla and
> 
> What's the meaning of vanilla?

Vanilla means an up-to-date (i.e., non-prehistoric) kernel from
kernel.org, without any "Value Added" patches from a distribution.

See: https://www.kernel.org/

Regards,

					- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>