Re: [PATCH] fs: add fincore(2) (mincore(2) for file descriptors)

[Jamie Lokier <jamie@shareable.org>]

Chris Frost wrote:
> We introduced this system call while modifying SQLite and the GIMP to
> request large prefetches for what would otherwise be non-sequential reads.
> As a macrobenchmark, we see a 125s SQLite query (72s system time) reduced
> to 75s (18s system time) by using fincore() instead of mincore(). This
> speedup of course varies by benchmark and benchmarks size; we've seen
> both minimal speedups and 1000x speedups. More on these benchmarks in the
> publication _Reducing Seek Overhead with Application-Directed Prefetching_
> in USENIX ATC 2009 and at http://libprefetch.cs.ucla.edu/.

My first thought was:

Why is calling fincore() and then issuing reads better than simply
calling readahead() on the same range?  I.e. why is readahead() (or
POSIX_FADV_WILLNEED) unsuitable to give the same result?  Or even
issuing lots of AIO requests.

I knew that I was missing something, so I read the paper ;-) I don't
fully understand it, but *think* that it says fincore() is used to
detect when the kernel is evicting pages faster than libprefetch had
planned for, implying memory pressure, so it adjusts its planning in
response.

Interesting idea, though I wonder if it wouldn't be even better to
have a direct way to ask the kernel "tell me when there is memory
pressure causing my file to be evicted".

-- Jamie