Is there a way to turn file caching off ?

Is there a way to turn file caching off ?

[Laurent Chavet]

Hi,

I'm running on a machine with 2GB of memory and dual PIII 550MHZ.
Just after boot with "nothing else running":
I run a program that almost like dd if=/dev/null of=/local/test
count=10000 bs=1000000
Except that there is a thread for reading and a thread for writing.
The program itself almost doesn't take any CPU.
What's going on is top showing:
    First cache grows to the size of RAM (2GB) with transfer rate
slowing down as the cache grows.
    Then the transfer rates drops a lot (2 to 3 time slower than the
drive capacity) and there is a very high CPU usage of system time (more
than a CPU) used by bdflush and kswapd (and some others like kupdated).

Of course my real application doesn't go from /dev/zero to file but it
still only does sequential access, and it seems that I pay a high price
for the file caching when I'm not using it at all.

Is there a way to turn file caching off, or at least limit its size ?

Thanks,

Laurent Chavet

RE: Is there a way to turn file caching off ?

[David Schwartz]

> Is there a way to turn file caching off, or at least limit its size ?
>
> Thanks,
>
> Laurent Chavet

	What benefit do you think you would get by limiting its size? All that
would do is ensure you hit the cache thrashing point sooner.

	DS

Re: Is there a way to turn file caching off ?

[Bjorn Wesen]

A similar phenomenon happens when you simply copy a file - file A is read
into the cache and file B is written to the cache, until the memory runs
out. Then both start to flush at the same time, creating a horrible
performance hit (especially if A and B are on the same disk :) 

I don't know a way to fix this except having the kernel correctly identify
the access pattern and optimize for it (i.e. if it recognizes that cache
pages are flushed in order to make room for more pages from the same
inode, then it's probably a suboptimal caching pattern and instead it
should probably increase the readahead and flush bigger chunks of pages at
the same time). I don't think anything can be done to the writing queue
(except maybe make the kernel understand that seek-time is more expensive
than transfer-time, so it does not schedule the read/writeing each odd
page..)

I'm still using 2.4.0 though so maybe this behaviour has been fixed to the
better in later kernels.. 

As a sidenote, try the same thing on an WinNT box and watch it die :) Like
unpacking a 1 GB file on a machine with 128 MB ram.. after it has unpacked
the first 100 MB's or so, performance drops to 1% or something..

-BW

On Tue, 17 Apr 2001, Laurent Chavet wrote:
>     First cache grows to the size of RAM (2GB) with transfer rate
> slowing down as the cache grows.
>     Then the transfer rates drops a lot (2 to 3 time slower than the
> drive capacity) and there is a very high CPU usage of system time (more
> than a CPU) used by bdflush and kswapd (and some others like kupdated).

Re: Is there a way to turn file caching off ?

[Jeremy Jackson]

Bjorn Wesen wrote:

> A similar phenomenon happens when you simply copy a file - file A is read
> into the cache and file B is written to the cache, until the memory runs
> out. Then both start to flush at the same time, creating a horrible

in this example only file B needs uses IO when being flushed; A's
buffers aren't dirty, so they're just 'forgotten'

>
> performance hit (especially if A and B are on the same disk :)
>

bdflush and 2.4's equivalent are tunable; if all you do is this copying
then optimise these parameters for this task.

>
> I don't know a way to fix this except having the kernel correctly identify
> the access pattern and optimize for it (i.e. if it recognizes that cache

currently all the kernel's heuristics are feed-back control loops.
what you are asking for is a feed-forward system: a way for the application
to tell kernel "I'm only reading this once, so after I'm done, throw it out
straight away"
and "I'm only writing this data, so after I'm done, start writing it out and
then forget it"

perhaps you could try mounting the destinatinon 'sync'
with dd at least you could specify block size,
there is also raw devices (no cache) mmaped files with madvise,
fdatasync()... lots of ways.

>
> pages are flushed in order to make room for more pages from the same
> inode, then it's probably a suboptimal caching pattern and instead it
> should probably increase the readahead and flush bigger chunks of pages at
> the same time). I don't think anything can be done to the writing queue
> (except maybe make the kernel understand that seek-time is more expensive
> than transfer-time, so it does not schedule the read/writeing each odd

there is the elevator seek mechanism...

>
> page..)
>
> I'm still using 2.4.0 though so maybe this behaviour has been fixed to the
> better in later kernels..
>
> As a sidenote, try the same thing on an WinNT box and watch it die :) Like
> unpacking a 1 GB file on a machine with 128 MB ram.. after it has unpacked
> the first 100 MB's or so, performance drops to 1% or something..

Isn't this due to the journaling filesystem?  All that seeking sure slows
things down...

>
>
> -BW
>
> On Tue, 17 Apr 2001, Laurent Chavet wrote:
> >     First cache grows to the size of RAM (2GB) with transfer rate
> > slowing down as the cache grows.
> >     Then the transfer rates drops a lot (2 to 3 time slower than the
> > drive capacity) and there is a very high CPU usage of system time (more
> > than a CPU) used by bdflush and kswapd (and some others like kupdated).
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

RE: Is there a way to turn file caching off ?

[David Schwartz]

	It's insignificant. If we assume the caching isn't helping, that means each
'non-recent' page access results in an IO operation. The cost of an IO
operation (a millisecond or more) is so much in comparison to the cost of a
primary cache miss (less than a microsecond) that avoiding a ferw primary
cache misses per IO isn't likely to make any measurable difference.

	DS

-----Original Message-----
From: lchavet@smo.av.com [mailto:lchavet@smo.av.com]On Behalf Of Laurent
Chavet
Sent: Wednesday, April 18, 2001 1:21 AM
To: David Schwartz
Cc: linux-kernel@vger.kernel.org
Subject: Re: Is there a way to turn file caching off ?


Since the speed already drops before even writing to disk, I was thinking
that for "large memory" the management of the tree that contain which pages
are cached becomes high (since the tree is large and probably doesn't fit in
primary cache).
So if you can limit the size of what you cache, you limit the size of the
tree -> you limit the time spent in the tree.
(by the way I'm using both 2.4.2 that come with RedHat 7.1 and 2.4.4 pre 3
and see the same thing).
Laurent

David Schwartz wrote:
> Is there a way to turn file caching off, or at least limit its size ?
>
> Thanks,
>
> Laurent Chavet
        What benefit do you think you would get by limiting its size? All
that
would do is ensure you hit the cache thrashing point sooner.
        DS
--
Laurent Chavet

Re: Is there a way to turn file caching off ?

[Andrea Arcangeli]

On Wed, Apr 18, 2001 at 11:21:46AM -0700, David Schwartz wrote:
> 
> 
> 	[..] If we assume the caching isn't helping [..]

If you know kernel data cache doesn't help your workload at all then you want
use O_DIRECT at least to save the CPU.  You can run 2.4.4pre3aa3 or apply the
rawio-3 patch and then the o_direct-2 patch on top of 2.4.4pre3 if you want
O_DIRECT support (it also fixes the O_SYNC wait for locked inode bug noticed by
Marcelo).

You will find detailed explanation on the O_DIRECT feature and where to
find the patches if you grep for subject O_DIRECT in the l-k list in the
messages of this month.

Andrea

Re: Is there a way to turn file caching off ?

[Helge Hafting]

Jeremy Jackson wrote:

> currently all the kernel's heuristics are feed-back control loops.
> what you are asking for is a feed-forward system: a way for the application
> to tell kernel "I'm only reading this once, so after I'm done, throw it out
> straight away"
> and "I'm only writing this data, so after I'm done, start writing it out and
> then forget it"
> 
This is hard to get right.  Sure - your unpack/copy program read once
and
writes once.  But the stuff might be used shortly thereafter by
another process.  For example:  I unpack a kernel tarball.  tar
knows it writes only once and might not need more than 5M to do
this as efficient as possible with my disks.  A lot of other cache
could be saved, fewer things swapped out.
But then I compile it.  Todays system ensures that lots of the source
is in memory already.  Limiting the caching to what tar needed
however will force the source to be read from disk once during
the compile - not what I want at all.  

A program may know its own access pattern, but it don't usually know
future access patterns.  Well, backing up the entire fs could benefit
from a something like this, you probably won't need the backup again
soon.  But this is hard to know in many other cases.

Helge Hafting

Re: Is there a way to turn file caching off ?

[Jeremy Jackson]

Helge Hafting wrote:

> Jeremy Jackson wrote:
>
> > currently all the kernel's heuristics are feed-back control loops.
> > what you are asking for is a feed-forward system: a way for the application
> > to tell kernel "I'm only reading this once, so after I'm done, throw it out
> > straight away"
> > and "I'm only writing this data, so after I'm done, start writing it out and
> > then forget it"
> >
> This is hard to get right.  Sure - your unpack/copy program read once
> and
> writes once.  But the stuff might be used shortly thereafter by
> another process.  For example:  I unpack a kernel tarball.  tar
> knows it writes only once and might not need more than 5M to do
> this as efficient as possible with my disks.  A lot of other cache
> could be saved, fewer things swapped out.
> But then I compile it.  Todays system ensures that lots of the source
> is in memory already.  Limiting the caching to what tar needed
> however will force the source to be read from disk once during
> the compile - not what I want at all.

They why would you tell tar not to use cache?  If you know what's happening
next you need to tell the system (feed-forward), not have it try to read your
mind.  I'm assuming your modified tar would have an option switch
to cause this behaviour, not be hard coded...

>
>
> A program may know its own access pattern, but it don't usually know
> future access patterns.  Well, backing up the entire fs could benefit

Yes, so a script that does the above wouldn't enable no cache mode
for written files.  The program doesn't know, but the encompasing
script (or person at console) does.

>
> from a something like this, you probably won't need the backup again
> soon.  But this is hard to know in many other cases.

Re: Is there a way to turn file caching off ?

[John Lenton]

On Thu, Apr 19, 2001 at 10:21:14AM +0200, Helge Hafting wrote:
> A program may know its own access pattern, but it don't usually know
> future access patterns.  Well, backing up the entire fs could benefit
> from a something like this, you probably won't need the backup again
> soon.  But this is hard to know in many other cases.

tar --please-leave-this-in-cache-pretty-please ?

-- 
John Lenton (john@grulic.org.ar) -- Random fortune:
Si los bugs te abruman, cierra tu Windows.