Drop caches - is this safe behavior?

Drop caches - is this safe behavior?

[mike]

I have a crontab running every 5 minutes on my servers now:

    echo 2 > /proc/sys/vm/drop_caches

Is this a safe thing to do? Am I risking any loss of data? It looks
like "3" might allow for that but from what I can understand 0-2 won't
lose data.

I was seeing some issues with my memory being taken up and thrown all
into "cached" and eventually starts swapping (not a lot, but a little)
- supposedly memory in "cached" is supposed to be available for new
stuff, but I swear it is not. I've tried a variety of things, and this
drop caches trick seems to make me feel quite comfortable seeing it be
free as in free physical RAM, not stuck in the cache.

So far it appears to be keeping my webservers' memory usage tolerable
and expected, as opposed to rampant and greedy. I haven't seen any
loss in functionality either. These servers get all their files (sans
local /var /etc stuff) from NFS, so I don't think a local memory-based
cache needs to be that important.

I've been trying to find more information on the drop_caches parameter
and its effects but it appears to be too new and not very widespread.
Any help is appreciated. Perhaps this is a safe behavior on a
non-primary file storage system like a webserver mounting NFS, but the
NFS server itself should not?

Thanks,
mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[Chris Snook]

mike wrote:
> I have a crontab running every 5 minutes on my servers now:
> 
>     echo 2 > /proc/sys/vm/drop_caches
> 
> Is this a safe thing to do? Am I risking any loss of data? It looks
> like "3" might allow for that but from what I can understand 0-2 won't
> lose data.
> 
> I was seeing some issues with my memory being taken up and thrown all
> into "cached" and eventually starts swapping (not a lot, but a little)
> - supposedly memory in "cached" is supposed to be available for new
> stuff, but I swear it is not. I've tried a variety of things, and this
> drop caches trick seems to make me feel quite comfortable seeing it be
> free as in free physical RAM, not stuck in the cache.

This widespread fallacy has been false for a long time.  The kernel will 
swap instead of dropping caches if it believes that doing so will 
improve performance.  It uses heuristics for this, and sometimes guesses 
wrong, but it's not a bad thing.  Consider the memory used in the 
initialization and shutdown routines for an application.  In normal 
operation, you're never using it, so it's much better to swap this out 
than to drop caches of a file you've actually accessed recently.  It is 
completely normal to use have a small amount of swap utilization for 
precisely this reason.  All you're doing by dropping caches is hurting 
performance, probably by a lot.  The drop_caches patch was resisted for 
a very long time because we knew people would use it to shoot themselves 
in the foot.  It should only be used for debugging or benchmarking.

> So far it appears to be keeping my webservers' memory usage tolerable
> and expected, as opposed to rampant and greedy. I haven't seen any
> loss in functionality either. These servers get all their files (sans
> local /var /etc stuff) from NFS, so I don't think a local memory-based
> cache needs to be that important.

"Rampant and greedy" is correct behavior, as long as it doesn't harm 
performance.  Usually it helps performance, but if it does harm 
performance, let us know, since that would be a bug we need to fix.  By 
dropping caches of NFS-backed files on multiple systems, you're moving 
the load from several underutilized systems to one heavily-utilized system.

> I've been trying to find more information on the drop_caches parameter
> and its effects but it appears to be too new and not very widespread.
> Any help is appreciated. Perhaps this is a safe behavior on a
> non-primary file storage system like a webserver mounting NFS, but the
> NFS server itself should not?

Safety aside, it's harmful no matter where you do it.  Forget about 
drop_caches.  Don't use it.  It wasn't meant for your use case.

If you think the system is doing the wrong thing (and it doesn't sound 
like it is) you should be tweaking the vm.swappiness sysctl.  The 
default is 60, but lower values will make it behave more like you think 
it should be behaving, though you'll still probably see a tiny bit of 
swap usage.  Of course, if your webservers are primarily serving up 
static content, you'll want a higher value, since swapping anonymous 
memory will leave more free for the pagecache you're primarily working with.

	-- Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[mike]

thanks for the reply... few comments

On 8/23/07, Chris Snook <csnook@redhat.com> wrote:

> This widespread fallacy has been false for a long time.  The kernel will
> swap instead of dropping caches if it believes that doing so will
> improve performance.  It uses heuristics for this, and sometimes guesses
> wrong, but it's not a bad thing.  Consider the memory used in the
> initialization and shutdown routines for an application.  In normal
> operation, you're never using it, so it's much better to swap this out
> than to drop caches of a file you've actually accessed recently.  It is
> completely normal to use have a small amount of swap utilization for
> precisely this reason.  All you're doing by dropping caches is hurting
> performance, probably by a lot.  The drop_caches patch was resisted for
> a very long time because we knew people would use it to shoot themselves
> in the foot.  It should only be used for debugging or benchmarking.

with this going right now, i have 0 megs of swap being used, tons of
physical memory available. basically what i would expect to see. i
have the PHP fastcgi engines killing themselves after only 50 requests
(aggressive) - its their effort to keep memory leaks away. i have
tried both aggressive and non-aggressive request numbers.

> "Rampant and greedy" is correct behavior, as long as it doesn't harm
> performance.  Usually it helps performance, but if it does harm
> performance, let us know, since that would be a bug we need to fix.  By
> dropping caches of NFS-backed files on multiple systems, you're moving
> the load from several underutilized systems to one heavily-utilized system.

it seems to affect performance. that's one of the reasons i am trying this out.

> Safety aside, it's harmful no matter where you do it.  Forget about
> drop_caches.  Don't use it.  It wasn't meant for your use case.
>
> If you think the system is doing the wrong thing (and it doesn't sound
> like it is) you should be tweaking the vm.swappiness sysctl.  The
> default is 60, but lower values will make it behave more like you think
> it should be behaving, though you'll still probably see a tiny bit of
> swap usage.  Of course, if your webservers are primarily serving up
> static content, you'll want a higher value, since swapping anonymous
> memory will leave more free for the pagecache you're primarily working with.

i have played with swappiness, i had it down to 10 at one point.

i have the drop_caches going every 5 minutes. the servers seem to be
running well. i had complaints after maybe 24 hours that things began
to lag on the web side. this has been running great so far.

i am also trying out a drop_caches with swappiness of 0 too on one
machine to see how that works.

it doesn't seem to be hurting anything currently. the NFS server is
performing with plenty of idle CPU, webservers have free memory, not
stuck up in swap that seems to not necessarily recycle nicely.

i'm trying deadline scheduler, the SLUB allocator, all sorts of things
here. sysctl configuration for tcp tweaks - with and without. nfs v3
and nfs v4. this actually seems to work though...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[Dave Kleikamp]

On Thu, 2007-08-23 at 21:36 -0400, Chris Snook wrote:

> If you think the system is doing the wrong thing (and it doesn't sound 
> like it is) you should be tweaking the vm.swappiness sysctl.  The 
> default is 60, but lower values will make it behave more like you think 
> it should be behaving, though you'll still probably see a tiny bit of 
> swap usage.  Of course, if your webservers are primarily serving up 
> static content, you'll want a higher value, since swapping anonymous 
> memory will leave more free for the pagecache you're primarily working with.

swappiness deals with page cache, whereas writing "2" to drop_caches
cleans out the inode and dentry caches.  Mike may be better off writing
a high number (say 10000) to /proc/sys/vm/vfs_cache_pressure.  This
would cause inode and dentry cache to be reclaimed sooner than other
memory.

-- 
David Kleikamp
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[Chris Snook]

mike wrote:
> thanks for the reply... few comments
> 
> On 8/23/07, Chris Snook <csnook@redhat.com> wrote:
> 
>> This widespread fallacy has been false for a long time.  The kernel will
>> swap instead of dropping caches if it believes that doing so will
>> improve performance.  It uses heuristics for this, and sometimes guesses
>> wrong, but it's not a bad thing.  Consider the memory used in the
>> initialization and shutdown routines for an application.  In normal
>> operation, you're never using it, so it's much better to swap this out
>> than to drop caches of a file you've actually accessed recently.  It is
>> completely normal to use have a small amount of swap utilization for
>> precisely this reason.  All you're doing by dropping caches is hurting
>> performance, probably by a lot.  The drop_caches patch was resisted for
>> a very long time because we knew people would use it to shoot themselves
>> in the foot.  It should only be used for debugging or benchmarking.
> 
> with this going right now, i have 0 megs of swap being used, tons of
> physical memory available. basically what i would expect to see.

You're expecting the wrong thing.  On any steady workload, free physical 
memory should only be just enough to satisfy emergency allocations, 
generally a few percent at most.  If any more than this is free, you're 
getting less performance than you would be if you had better (higher) 
utilization.  There are fleetingly rare exceptions to this, all of them 
bugs that have been or should be fixed.

> i
> have the PHP fastcgi engines killing themselves after only 50 requests
> (aggressive) - its their effort to keep memory leaks away. i have
> tried both aggressive and non-aggressive request numbers.

Okay, so it sounds like you have an extremely unsteady load, due to this 
leak/restart cycle.  That's the problem you need to be fixing.  The 
kernel is behaving correctly.

>> "Rampant and greedy" is correct behavior, as long as it doesn't harm
>> performance.  Usually it helps performance, but if it does harm
>> performance, let us know, since that would be a bug we need to fix.  By
>> dropping caches of NFS-backed files on multiple systems, you're moving
>> the load from several underutilized systems to one heavily-utilized system.
> 
> it seems to affect performance. that's one of the reasons i am trying this out.

Can you quantify this, perhaps with sar data?

>> Safety aside, it's harmful no matter where you do it.  Forget about
>> drop_caches.  Don't use it.  It wasn't meant for your use case.
>>
>> If you think the system is doing the wrong thing (and it doesn't sound
>> like it is) you should be tweaking the vm.swappiness sysctl.  The
>> default is 60, but lower values will make it behave more like you think
>> it should be behaving, though you'll still probably see a tiny bit of
>> swap usage.  Of course, if your webservers are primarily serving up
>> static content, you'll want a higher value, since swapping anonymous
>> memory will leave more free for the pagecache you're primarily working with.
> 
> i have played with swappiness, i had it down to 10 at one point.

Can you quantify the impact it had?  Perhaps your workload is one we 
could use to inform slightly more intelligent self-tuning in the VM, but 
we'd need hard numbers before we could even begin such analysis.

> i have the drop_caches going every 5 minutes. the servers seem to be
> running well. i had complaints after maybe 24 hours that things began
> to lag on the web side. this has been running great so far.

If you need to do it every 5 minutes, that must be one hell of a bad 
memory leak in your application.  I've seen arguably legitimate uses of 
drop_caches to clean up after backup jobs.  They fire exactly once, at 
the end of the job.  Pagecache is only filling back up because you're 
actually using those files, so all of the data you're getting rid of is 
data that has been used in the last 5 minutes.

If your complaints are happening after 24 hours, then drop caches once 
every 12 hours, at non-peak times.  It's still an ugly hack, but it's a 
less damaging ugly hack.  If you're running out of anything, it's 
probably *lower zone* memory, or *contiguous* memory for NFS operations 
and DMA buffers.  If you're on i386, vm.lower_zone_protection=100 will 
mitigate this.

> i am also trying out a drop_caches with swappiness of 0 too on one
> machine to see how that works.
> 
> it doesn't seem to be hurting anything currently. the NFS server is
> performing with plenty of idle CPU, webservers have free memory, not
> stuck up in swap that seems to not necessarily recycle nicely.

There's no such thing as "stuck up in swap".  Swap is an optimization, 
and the kernel has chosen to use swap because it believes that doing so 
will free memory for more important pagecache.  Swap doesn't "recycle" 
back into RAM unless someone is actually using it.  The fact that it 
remains in swap *proves* that it belongs there.

> i'm trying deadline scheduler, the SLUB allocator, all sorts of things
> here. sysctl configuration for tcp tweaks - with and without. nfs v3
> and nfs v4. this actually seems to work though...

Sure, it works, but it's not efficient, and it should never be 
necessary.  If it truly is necessary, we need to fix the VM.

	-- Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[Chris Snook]

Dave Kleikamp wrote:
> On Thu, 2007-08-23 at 21:36 -0400, Chris Snook wrote:
> 
>> If you think the system is doing the wrong thing (and it doesn't sound 
>> like it is) you should be tweaking the vm.swappiness sysctl.  The 
>> default is 60, but lower values will make it behave more like you think 
>> it should be behaving, though you'll still probably see a tiny bit of 
>> swap usage.  Of course, if your webservers are primarily serving up 
>> static content, you'll want a higher value, since swapping anonymous 
>> memory will leave more free for the pagecache you're primarily working with.
> 
> swappiness deals with page cache, whereas writing "2" to drop_caches
> cleans out the inode and dentry caches.  Mike may be better off writing
> a high number (say 10000) to /proc/sys/vm/vfs_cache_pressure.  This
> would cause inode and dentry cache to be reclaimed sooner than other
> memory.
> 

Thanks, I was confusing this with dropping pagecache.

Mike --

	Try Dave's suggestion to increase vm.vfs_cache_pressure.  drop_pages 
should never be needed, regardless of which caches you're dropping.

	-- Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[mike]

On 8/23/07, Chris Snook <csnook@redhat.com> wrote:
> Mike --
>
>        Try Dave's suggestion to increase vm.vfs_cache_pressure.  drop_pages
> should never be needed, regardless of which caches you're dropping.
>
>        -- Chris
>

thanks all. i will try it on one of the machines and see how it performs.

this is an opteron 1.8ghz (amd64), ubuntu, latest stable linux kernel,
3 gigs of ram (just FYI) - SATA disk.

i thought i'd do it every 5 minutes not because of a horrible memory
leak that fast, but figured "why not just free up all RAM as often as
possible"

when you said "sar" are you talking about this:

atsar - system activity reporter
Description: system activity reporter
 Monitor system resources such as CPU, network, memory & disk I/O, and
 record data for later analysis

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[Chris Snook]

mike wrote:
> On 8/23/07, Chris Snook <csnook@redhat.com> wrote:
>> Mike --
>>
>>        Try Dave's suggestion to increase vm.vfs_cache_pressure.  drop_pages
>> should never be needed, regardless of which caches you're dropping.
>>
>>        -- Chris
>>
> 
> thanks all. i will try it on one of the machines and see how it performs.
> 
> this is an opteron 1.8ghz (amd64), ubuntu, latest stable linux kernel,
> 3 gigs of ram (just FYI) - SATA disk.
> 
> i thought i'd do it every 5 minutes not because of a horrible memory
> leak that fast, but figured "why not just free up all RAM as often as
> possible"

I think the caches you had in mind were the ones that would be dropped 
by echoing '1' into /proc/sys/vm/drop_caches, not the ones that would be 
dropped by echoing '2' into it.  If you were dropping pagecache every 
five minutes, it would kill your performance as you described.  As for 
the question of safety, '3' should also be safe, but terrible for 
performance, as it does all the harm of '1', plus some.

> when you said "sar" are you talking about this:
> 
> atsar - system activity reporter
> Description: system activity reporter
>  Monitor system resources such as CPU, network, memory & disk I/O, and
>  record data for later analysis

I'm not familiar with the "atsar" implementation, but it appears to be 
an alternate implementation of the same thing.  It's an excellent tool 
for long-term workload profiling.

	-- Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[mike]

On 8/23/07, Chris Snook <csnook@redhat.com> wrote:
> I think the caches you had in mind were the ones that would be dropped
> by echoing '1' into /proc/sys/vm/drop_caches, not the ones that would be
> dropped by echoing '2' into it.  If you were dropping pagecache every
> five minutes, it would kill your performance as you described.  As for
> the question of safety, '3' should also be safe, but terrible for
> performance, as it does all the harm of '1', plus some.

actually right now the performance seems to be good - using "2"

i'm willing to try "1" as well, as well as try the cache pressure one.
i don't really know what caches i am clearing, but it seems that i get
bottlenecked by something. restarting my webserver/php engines usually
clears it up, so it seems like it is a buildup of something - and it
always seems to be when memory is tight...

> I'm not familiar with the "atsar" implementation, but it appears to be
> an alternate implementation of the same thing.  It's an excellent tool
> for long-term workload profiling.

actually, this might be a better method - is there any way to view the
contents of the cache? or figure out what exactly is sitting in
there/why my machine thinks it needs to cache 2 gigs of files so
quickly?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Drop caches - is this safe behavior?

[Chris Snook]

mike wrote:
> On 8/23/07, Chris Snook <csnook@redhat.com> wrote:
>> I think the caches you had in mind were the ones that would be dropped
>> by echoing '1' into /proc/sys/vm/drop_caches, not the ones that would be
>> dropped by echoing '2' into it.  If you were dropping pagecache every
>> five minutes, it would kill your performance as you described.  As for
>> the question of safety, '3' should also be safe, but terrible for
>> performance, as it does all the harm of '1', plus some.
> 
> actually right now the performance seems to be good - using "2"

That's because you're not actually dropping that much.

> i'm willing to try "1" as well, as well as try the cache pressure one.
> i don't really know what caches i am clearing, but it seems that i get
> bottlenecked by something. restarting my webserver/php engines usually
> clears it up, so it seems like it is a buildup of something - and it
> always seems to be when memory is tight...

'1' drops pagecache, which is the kernel's cache of file data.  This is what you 
see under "cached" in the output of free.  '2' drops VFS cache, which is the 
kernel's cache of file *metadata*.  All of this appears under "used" in the 
output of free.  '3' drops both.  I think what you actually want is '1' or '3', 
though I assure you it would hurt performance considerably if you do either of 
those.

Turning up vm.vfs_cache_pressure instead of echoing '2' into drop_caches will 
cause the system to automatically free up more of that cache, but only when 
necessary.  It's much less disruptive to the system.

>> I'm not familiar with the "atsar" implementation, but it appears to be
>> an alternate implementation of the same thing.  It's an excellent tool
>> for long-term workload profiling.
> 
> actually, this might be a better method - is there any way to view the
> contents of the cache? or figure out what exactly is sitting in
> there/why my machine thinks it needs to cache 2 gigs of files so
> quickly?

If your system is caching 2 GB of files, that's because it's actually using 2 GB 
of files.  It *might* be useful to do a drop_caches *once* after the system 
boots and services start to clear out anything that's only used at boot, but 
doing this during normal operation is just shooting yourself in the foot.

Also, check out how your applications are using temporary files.  If files are 
deleted and aren't in use by any other processes, their pagecache gets dropped. 
  If your apps are being lazy and waiting for tmpwatch to clean up their 
garbage, it's sitting around in pagecache because nobody has informed the kernel 
that they're no longer needed.

	-- Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>