Re: Very aggressive memory reclaim

Re: Very aggressive memory reclaim

[Steven Rostedt]

[ Add Cc's of those that may help you ]

-- Steve

On Mon, Mar 28, 2011 at 08:39:29PM +0400, John Lepikhin wrote:
> Hello,
> 
> I use high-loaded machine with 10M+ inodes inside XFS, 50+ GB of
> memory, intensive HDD traffic and 20..50 forks per second. Vanilla
> kernel 2.6.37.4. The problem is that kernel frees memory very
> aggressively.
> 
> For example:
> 
> 25% of memory is used by processes
> 50% for page caches
> 7% for slabs, etc.
> 18% free.
> 
> That's bad but works. After few hours:
> 
> 25% of memory is used by processes
> 62% for page caches
> 7% for slabs, etc.
> 5% free.
> 
> Most of files are cached, works perfectly. This is the moment when
> kernel decides to free some memory. After memory reclaim:
> 
> 25% of memory is used by processes
> 25% for page caches(!)
> 7% for slabs, etc.
> 43% free(!)
> 
> Page cache is dropped, server becomes too slow. This is the beginning
> of new cycle.
> 
> I didn't found any huge mallocs at that moment. Looks like because of
> large number of small mallocs (forks) kernel have pessimistic forecast
> about future memory usage and frees too much memory. Is there any
> options of tuning this? Any other variants?
> 
> Thanks!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Very aggressive memory reclaim

[John Lepikhin]

2011/3/28 Jeffrey Hundstad <jeffrey.hundstad@mnsu.edu>:

> I'd take a look here:
> http://www.linuxinsight.com/proc_sys_vm_hierarchy.html

Yes, I already played with dirty_*, min_free_kbytes (3000kb),
swappiness (0..100), vfs_cache_pressure (1..200) and zone_reclaim_mode
(currently 0). Other parameters are set to defaults.

By the way, there is no swap enabled. Instead of just dropping 50% of
page caches, kernel was intensively swapping then there was a swap
device.

Re: Very aggressive memory reclaim

[Wu Fengguang]

Hi John,

On Mon, Mar 28, 2011 at 10:50:56PM +0400, John Lepikhin wrote:
> 2011/3/28 Jeffrey Hundstad <jeffrey.hundstad@mnsu.edu>:
> 
> > I'd take a look here:
> > http://www.linuxinsight.com/proc_sys_vm_hierarchy.html
> 
> Yes, I already played with dirty_*, min_free_kbytes (3000kb),
> swappiness (0..100), vfs_cache_pressure (1..200) and zone_reclaim_mode
> (currently 0). Other parameters are set to defaults.
> 
> By the way, there is no swap enabled. Instead of just dropping 50% of
> page caches, kernel was intensively swapping then there was a swap
> device.

Is your memory usage balanced across the nodes? You can check it via
/sys/devices/system/node/node*/meminfo.

Are there lots of high-order memory allocations?  /proc/buddyinfo will
disclose some of them.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>