Re: Disk Cache, Was: O_DIRECT question

[Michael Tokarev <mjt@tls.msk.ru>]

Zan Lynx wrote:
> On Sat, 2007-01-13 at 00:03 +0300, Michael Tokarev wrote:
> [snip]
>> And sure thing, withOUT O_DIRECT, the whole system is almost dead under this
>> load - because everything is thrown away from the cache, even caches of /bin
>> /usr/bin etc... ;)  (For that, fadvise() seems to help a bit, but not alot).
> 
> One thing that I've been using, and seems to work well, is a customized
> version of the readahead program several distros use during boot up.

[idea to lock some (commonly-used) cache pages in memory]

> Something like that could keep your system responsive no matter what the
> disk cache is doing otherwise.

Unfortunately it's not.  Sure, things like libc.so etc will be force-cached
and will start fast.  But not my data files and other stuff (what an
unfortunate thing: memory usually is smaller in size than disks ;)

I can do usual work without noticing something's working with the disks
intensively, doing O_DIRECT I/O.  For example, I can run large report on
a database, which requires alot of disk I/O, and run a kernel compile at
the same time.  Sure, disk access is alot slower, but disk cache helps alot,
too.  My kernel compile will not be much slower than usual.  But if I'll
turn O_DIRECT off, the compile will take ages to finish.  *And* the report
running, too!  Because the system tries hard to cache the WRONG pages!
(yes I remember fadvise &Co - which aren't used by the database(s) currently,
and quite alot of words has been said about that, too;  I also noticied it's
slower as well, at least currently.)

/mjt