Ability to limit or disable page caching?

Ability to limit or disable page caching?

[Kevin Burton]

I'm running into a VERY frustrating bug with page caching and swap.

I'm going to write a long post on this and forward it to the list.

In the mean time is it possible to disable the page cache or limit it
to say 10% of system memory?

I haven't found any way to do this yet.

You could do this with RHEL 4.x but I don't recall this ever making it
into 2.6 :-/

Feedback appreciated.

-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[FD Cami]

Hi,

On Mon, 5 May 2008 00:12:56 -0700
"Kevin Burton" <burton@tailrank.com> wrote:

> In the mean time is it possible to disable the page cache or limit it
> to say 10% of system memory?

You might want to look at /proc/sys/vm/ for tunables.
/proc/sys/vm/vfs_cache_pressure and /proc/sys/vm/swappiness are probably
what you are looking for.
See this article for an explanation of swappiness :
http://www.leap-cf.org/oldarchive/2004-May/038622.html

> You could do this with RHEL 4.x but I don't recall this ever making it
> into 2.6 :-/

AFAIK RHEL 4.x runs 2.6.9.

Cheers

FDC

Re: Ability to limit or disable page caching?

[Kevin Burton]

>> In the mean time is it possible to disable the page cache or limit it
>> to say 10% of system memory?
>
> You might want to look at /proc/sys/vm/ for tunables.
> /proc/sys/vm/vfs_cache_pressure and /proc/sys/vm/swappiness are probably
> what you are looking for.

I haven't played with vfs_cache_pressure so I'll take a look at this.

Unfortunately, swappiness doesn't work as intended.

> See this article for an explanation of swappiness :
> http://www.leap-cf.org/oldarchive/2004-May/038622.html
>
>> You could do this with RHEL 4.x but I don't recall this ever making it
>> into 2.6 :-/
>
> AFAIK RHEL 4.x runs 2.6.9.

OK... maybe it was 3.x .. I might be getting confused with their
kernel release numbers.

Kevin

-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[FD Cami]

On Mon, 5 May 2008 11:34:24 -0700
"Kevin Burton" <burtonator@gmail.com> wrote:

> >> In the mean time is it possible to disable the page cache or limit it
> >> to say 10% of system memory?
> >
> > You might want to look at /proc/sys/vm/ for tunables.
> > /proc/sys/vm/vfs_cache_pressure and /proc/sys/vm/swappiness are probably
> > what you are looking for.
> 
> I haven't played with vfs_cache_pressure so I'll take a look at this.
> 
> Unfortunately, swappiness doesn't work as intended.

Could you explain the original problem ? That way we might help you better.

Best,

Francois

Re: Ability to limit or disable page caching?

[Kevin Burton]

Yeah... I wanted to explain it in detail in a longer post.  If
necessary I'll do that as well.

I have a database process that's using 3G of memory. Then I have a
secondary process that's using about 500MB of memory.  This leaves
about 500MB free for anything else. The box has 4GB of memory total...
No swap is needed.

... swappiness is set to zero.

I've verified that overcommit or other issues aren't getting in the
way as I can run the box without swap.  This isn't possible when in
production though.

What's happening is that the kernel is deciding to swap out some
memory to disk in the (false) belief that it can free some up for
cache.

The problem is my database is already using its own cache so these are
conflicting philosophies here which make Linux perform in a
pathological way..

We're using O_DIRECT for our database which helps reduce the problem a
bit...... O_DIRECT bypasses the page cache and does all IO directly.
This means there isn't much pressure on the page cache and pages
aren't swapped out to disk.  However, MySQL still needs to do reads on
misc files which aren't O_DIRECT so I end up swapping but at a much
slower rate.

What I want to do is either disable the page cache entirely or just
tell the OS to cache at max 10% of the available memory.

Thoughts?

Kevin

On Mon, May 5, 2008 at 12:28 PM, FD Cami <francois.cami@free.fr> wrote:
> On Mon, 5 May 2008 11:34:24 -0700
> "Kevin Burton" <burtonator@gmail.com> wrote:
>
>> >> In the mean time is it possible to disable the page cache or limit it
>> >> to say 10% of system memory?
>> >
>> > You might want to look at /proc/sys/vm/ for tunables.
>> > /proc/sys/vm/vfs_cache_pressure and /proc/sys/vm/swappiness are probably
>> > what you are looking for.
>>
>> I haven't played with vfs_cache_pressure so I'll take a look at this.
>>
>> Unfortunately, swappiness doesn't work as intended.
>
> Could you explain the original problem ? That way we might help you better.
>
> Best,
>
> Francois
>



-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[Andi Kleen]

"Kevin Burton" <burton@tailrank.com> writes:
>
> What I want to do is either disable the page cache entirely or just
> tell the OS to cache at max 10% of the available memory.

FYI page cache includes the memory your program uses. No page 
cache would mean no user space, 10% would mean user space uses only
10%. One way that would fulfil your request literally is to boot with 
mem=<10% of your ram>, but I guess you don't really want that.

If you don't want your application to be swapped at all you
should probably investigate mlock(2)/mlockall(2)

-Andi

Re: Ability to limit or disable page caching?

[Kevin Burton]

We're actually running mlock which is native to MySQL.... of course I
didn't think about whether our secondary process is paging not our
MySQL process.

I'm going to have to look into that and figure out if it's possible to
figure out what apps are being paged.

Kevin

On Mon, May 5, 2008 at 1:10 PM, Andi Kleen <andi@firstfloor.org> wrote:
> "Kevin Burton" <burton@tailrank.com> writes:
>>
>> What I want to do is either disable the page cache entirely or just
>> tell the OS to cache at max 10% of the available memory.
>
> FYI page cache includes the memory your program uses. No page
> cache would mean no user space, 10% would mean user space uses only
> 10%. One way that would fulfil your request literally is to boot with
> mem=<10% of your ram>, but I guess you don't really want that.
>
> If you don't want your application to be swapped at all you
> should probably investigate mlock(2)/mlockall(2)
>
> -Andi
>



-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[Kevin Burton]

Though mlockall isn't going to fix this problem because the OOM killer
will still kill it if it runs out of memory.

I want to preserve the scenario where we go 10 bytes over memory and
the kernel just swaps out those 10 bytes.

Kevin

On Mon, May 5, 2008 at 1:28 PM, Kevin Burton <burton@tailrank.com> wrote:
> We're actually running mlock which is native to MySQL.... of course I
> didn't think about whether our secondary process is paging not our
> MySQL process.


-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[Kevin Burton]

... and here's another point I don't fully grok:

http://www.westnet.com/~gsmith/content/linux-pdflush.htm

"By default, Linux will aggressively swap processes out of physical
memory onto disk in order to keep the disk cache as large as possible.
This means that pages that haven't been used recently will be pushed
into swap long before the system even comes close to running out of
memory, which is an unexpected behavior compared to some operating
systems. The /proc/sys/vm/swappiness parameter controls how aggressive
Linux is in this area. "

...

"A value of 0 will avoid ever swapping out just for caching space.
Using 100 will always favor making the disk cache bigger. Most
distributions set this value to be 60, tuned toward moderately
aggressive swapping to increase disk cache. "

..... but we're running on zero and this machine still has 500MB of
free memory.  Why is it swapping?

Kevin

On Mon, May 5, 2008 at 1:31 PM, Kevin Burton <burton@tailrank.com> wrote:
> Though mlockall isn't going to fix this problem because the OOM killer
> will still kill it if it runs out of memory.
>
> I want to preserve the scenario where we go 10 bytes over memory and
> the kernel just swaps out those 10 bytes.
>
> Kevin
>
> On Mon, May 5, 2008 at 1:28 PM, Kevin Burton <burton@tailrank.com> wrote:
>> We're actually running mlock which is native to MySQL.... of course I
>> didn't think about whether our secondary process is paging not our
>> MySQL process.
>
>
> --
> Founder/CEO Tailrank.com
> Location: San Francisco, CA
> AIM/YIM: sfburtonator
> Skype: burtonator
> Work: http://spinn3r.com and http://tailrank.com
> Blog: http://feedblog.org
> Cell: 415-637-8078
> Fax: 1-415-358-419 PIN: 0092
>



-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[Andi Kleen]

Kevin Burton wrote:
> ... and here's another point I don't fully grok:

I would suggest you try to understand what I wrote in my first answer.
>From your replies you didn't seem to.

-Andi

Re: Ability to limit or disable page caching?

[Kevin Burton]

Grok it now.. It seems counterintuitive for both to be included.

Kevin

On Mon, May 5, 2008 at 5:23 PM, Andi Kleen <andi@firstfloor.org> wrote:
> Kevin Burton wrote:
>  > ... and here's another point I don't fully grok:
>
>  I would suggest you try to understand what I wrote in my first answer.
>  From your replies you didn't seem to.
>
>  -Andi
>
>



-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[Bernd Eckenfels]

In article <878wyocy0v.fsf@basil.nowhere.org> you wrote:
> If you don't want your application to be swapped at all you
> should probably investigate mlock(2)/mlockall(2)

Maybe this is similiar to the "well known" mysql dirty buffer problem?

It looks strange to me that sysadmins are forced to swap to ramdisks:

http://blogs.smugmug.com/don/2008/05/01/mysql-and-the-linux-swap-problem/

Gruss
Bernd

Re: Ability to limit or disable page caching?

[Rik van Riel]

On Mon, 5 May 2008 12:42:31 -0700
"Kevin Burton" <burton@tailrank.com> wrote:

> What's happening is that the kernel is deciding to swap out some
> memory to disk in the (false) belief that it can free some up for
> cache.
> 
> The problem is my database is already using its own cache so these are
> conflicting philosophies here which make Linux perform in a
> pathological way..

One of the problems is that the process pages (anonymous memory) and
page cache pages live on the same LRU, so the kernel cannot always
easily find the page cache pages when it is trying to evict something.

Once that is fixed, and replacement is biased towards evicting page
cache pages, the system may do the right thing by itself.

I realize that this is no quick fix for your issue, but I am working
on a split LRU patch series to make sure Linux does the right thing in
the future.

You can find the latest patch at http://people.redhat.com/riel/splitvm/
and info on the design at http://linux-mm.org/PageReplacementDesign

> What I want to do is either disable the page cache entirely or just
> tell the OS to cache at max 10% of the available memory.

I have no such tunable in my code (yet), because I would like the
kernel to do the right thing automatically.  I will post a 2.6.25 based
kernel RPM for Fedora 9 soon with the split LRU patch series applied.
If you feel like testing/breaking it, I would be interested to see if
it does indeed do the right thing for your workload or if it needs more
tuning.

-- 
All Rights Reversed

Re: Ability to limit or disable page caching?

[Kevin Burton]

> One of the problems is that the process pages (anonymous memory) and
> page cache pages live on the same LRU, so the kernel cannot always
> easily find the page cache pages when it is trying to evict something.

Ah... that makes sense.

> Once that is fixed, and replacement is biased towards evicting page
> cache pages, the system may do the right thing by itself.

Yeah.  That seems like it...

> I have no such tunable in my code (yet), because I would like the
> kernel to do the right thing automatically.  I will post a 2.6.25 based
> kernel RPM for Fedora 9 soon with the split LRU patch series applied.

With the same patch you linked to above?  No new code..

We're on Debian so I need to dive in and see how easy it would be for
us to compile our own kernel.

I've done it in the past but I've avoided running anything non-stock
for about 2 years now...

We're on 2.6.24 though but that's not a big issue.

> If you feel like testing/breaking it, I would be interested to see if
> it does indeed do the right thing for your workload or if it needs more
> tuning.

Well we'd be deploying it in product :).. what's the ETA into making
it into the official kernel?

Kevin


-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[Rik van Riel]

On Mon, 5 May 2008 14:00:51 -0700
"Kevin Burton" <burton@tailrank.com> wrote:

> > I have no such tunable in my code (yet), because I would like the
> > kernel to do the right thing automatically.  I will post a 2.6.25 based
> > kernel RPM for Fedora 9 soon with the split LRU patch series applied.
> 
> With the same patch you linked to above?  No new code..

Same patch, rediffed against 2.6.25.

> > If you feel like testing/breaking it, I would be interested to see if
> > it does indeed do the right thing for your workload or if it needs more
> > tuning.
> 
> Well we'd be deploying it in product :).. what's the ETA into making
> it into the official kernel?

The code seems to work right. What is left now is convincing Andrew Morton
and Linus Torvalds to merge it.

Getting good test results with reasonable workloads where the vanilla
kernel does the wrong thing (eg. your workload) is probably the most
important factor for Andrew and Linus in deciding whether or not to
merge the code. 

-- 
All Rights Reversed

Re: Ability to limit or disable page caching?

[Kevin Burton]

Note that this is part of a larger thread:

http://blogs.smugmug.com/don/2008/05/01/mysql-and-the-linux-swap-problem/

...

and my older post on the subject:

http://feedblog.org/2007/09/29/using-o_direct-on-linux-and-innodb-to-fix-swap-insanity/

mlockall DOES help solve this problem but then MySQL is forced to stay
in memory.

If you have too many connections to the server and it mallocs more
memory and the system can't page anything else it will OOM kill your
mysqld process.

It's MUCH better to have MySQL dip 100MB into swap first....

Kevin

On Mon, May 5, 2008 at 3:08 PM, Rik van Riel <riel@redhat.com> wrote:
> On Mon, 5 May 2008 14:00:51 -0700
> "Kevin Burton" <burton@tailrank.com> wrote:
>
>> > I have no such tunable in my code (yet), because I would like the
>> > kernel to do the right thing automatically.  I will post a 2.6.25 based
>> > kernel RPM for Fedora 9 soon with the split LRU patch series applied.
>>
>> With the same patch you linked to above?  No new code..
>
> Same patch, rediffed against 2.6.25.
>
>> > If you feel like testing/breaking it, I would be interested to see if
>> > it does indeed do the right thing for your workload or if it needs more
>> > tuning.
>>
>> Well we'd be deploying it in product :).. what's the ETA into making
>> it into the official kernel?
>
> The code seems to work right. What is left now is convincing Andrew Morton
> and Linus Torvalds to merge it.
>
> Getting good test results with reasonable workloads where the vanilla
> kernel does the wrong thing (eg. your workload) is probably the most
> important factor for Andrew and Linus in deciding whether or not to
> merge the code.
>
> --
> All Rights Reversed
>



-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

split LRU kernel RPM

[Rik van Riel]

On Mon, 5 May 2008 14:00:51 -0700
"Kevin Burton" <burton@tailrank.com> wrote:

> > I have no such tunable in my code (yet), because I would like the
> > kernel to do the right thing automatically.  I will post a 2.6.25 based
> > kernel RPM for Fedora 9 soon with the split LRU patch series applied.

I have created a 2.6.25 based split LRU kernel RPM for Fedora 9:

http://people.redhat.com/riel/splitvm/

The srpm and x86-64 RPMs are there.  The i686 kernel RPM is
still compiling, I will upload it tomorrow morning.

-- 
All rights reversed.

Re: Ability to limit or disable page caching?

[Chris Friesen]

Rik van Riel wrote:
> On Mon, 5 May 2008 12:42:31 -0700
> "Kevin Burton" <burton@tailrank.com> wrote:

>>What I want to do is either disable the page cache entirely or just
>>tell the OS to cache at max 10% of the available memory.

> I have no such tunable in my code (yet), because I would like the
> kernel to do the right thing automatically.

Given that the kernel doesn't currently do the right thing, what about 
providing a tuneable as a temporary fix for people that are currently 
having problems?

Or are you concerned that they will then not be as willing to test the 
automatic behaviour?

Chris

Re: Ability to limit or disable page caching?

[Kevin Burton]

I'd like to be able to disable the cache altogether.....  It seems
like an easy tunable to add and might help benefit future testing.


-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092

Re: Ability to limit or disable page caching?

[Rik van Riel]

On Mon, 5 May 2008 14:32:41 -0700
"Kevin Burton" <burton@tailrank.com> wrote:

> I'd like to be able to disable the cache altogether.....  It seems
> like an easy tunable to add and might help benefit future testing.

Executables, libraries, shared memory segments, directories and all kinds
of other things live in the page cache.  You cannot disable it without
breaking the most basic functionality.

-- 
All Rights Reversed

Re: Ability to limit or disable page caching?

[Tomasz Kłoczko]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 970 bytes --]

On Mon, 5 May 2008, FD Cami wrote:

> On Mon, 5 May 2008 11:34:24 -0700
> "Kevin Burton" <burtonator@gmail.com> wrote:
>
>>>> In the mean time is it possible to disable the page cache or limit it
>>>> to say 10% of system memory?
>>>
>>> You might want to look at /proc/sys/vm/ for tunables.
>>> /proc/sys/vm/vfs_cache_pressure and /proc/sys/vm/swappiness are probably
>>> what you are looking for.
>>
>> I haven't played with vfs_cache_pressure so I'll take a look at this.
>>
>> Unfortunately, swappiness doesn't work as intended.
>
> Could you explain the original problem ? That way we might help you better.

Probably it is something like this:

http://blogs.smugmug.com/don/2008/05/01/mysql-and-the-linux-swap-problem/

kloczek
-- 
-----------------------------------------------------------
*Ludzie nie mają problemów, tylko sobie sami je stwarzają*
-----------------------------------------------------------
Tomasz Kłoczko | *e-mail: kloczek@rudy.mif.pg.gda.pl*

Re: Ability to limit or disable page caching?

[Kevin Burton]

.
>> /proc/sys/vm/vfs_cache_pressure and /proc/sys/vm/swappiness are probably
>> what you are looking for.
>
> I haven't played with vfs_cache_pressure so I'll take a look at this.
>
> Unfortunately, swappiness doesn't work as intended.

OK....... I tried again and no such luck unfortunately.

Set vfs_cache_pressure to 0 and 200 (default is zero) and no luck either way.

A value of 0 us used for swappiness ....

Any other ideas...

The box has enough memory to do this as I can run it without swap....

Kevin

-- 
Founder/CEO Tailrank.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com and http://tailrank.com
Blog: http://feedblog.org
Cell: 415-637-8078
Fax: 1-415-358-419 PIN: 0092