Re: Lockless page cache test results

[Jens Axboe <axboe@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 2311 bytes --]

On Wed, Apr 26 2006, Andrew Morton wrote:
> Jens Axboe <axboe@suse.de> wrote:
> >
> > With a 16-page gang lookup in splice, the top profile for the 4-client
> > case (which is now at 4GiB/sec instead of 3) are:
> > 
> > samples  %        symbol name
> > 30396    36.7217  __do_page_cache_readahead
> > 25843    31.2212  find_get_pages_contig
> > 9699     11.7174  default_idle
> 
> __do_page_cache_readahead() should use gang lookup.  We never got around to
> that, mainly because nothing really demonstrated a need.
> 
> It's a problem that __do_page_cache_readahead() is being called at all -
> with everything in pagecache we should be auto-turning-off readahead.  This
> happens because splice is calling the low-level do_pagecache_readahead(). 
> If you convert it to use page_cache_readahead(), that will all vanish if
> readahead is working right.

You are right, I modified it to use page_cache_readahead() and that
looks a lot better for the vanilla kernel. Here's a new graph of
2.6.17-rc3 and 2.6.17-rc3-lockless. Both base kernels are the splice
branch, so it contains the contig lookup change as well.

I'm still doing only up to nr_cpus clients, as numbers start to
fluctuate above that.

Things look pretty bad for the lockless kernel though, Nick any idea
what is going on there? The splice change is pretty simple, see the top
three patches here:

http://brick.kernel.dk/git/?p=linux-2.6-block.git;a=shortlog;h=splice

The top profile for the 4-client case looks like this:

samples  %        symbol name
77955    77.7141  find_get_pages_contig
8034      8.0092  splice_to_pipe
5092      5.0763  default_idle
4330      4.3166  do_splice_to
1323      1.3189  sys_splice

where vanilla is nowhere near that bad. I added a third graph, which is
lockless with the top patch backed out again so it's using
find_get_page() for each page. The top profile then looks like this for
the 4-client case:

samples  %        symbol name
10685    39.2730  default_idle
4120     15.1432  find_get_page
2608      9.5858  sys_splice
1729      6.3550  radix_tree_lookup_slot
1708      6.2778  splice_from_pipe
1071      3.9365  splice_to_pipe

Finally, I'll just note that the find_get_pages_contig() was modified
for the lockless kernel (read_lock -> spin_lock), so it isn't something
silly :-)

-- 
Jens Axboe


[-- Attachment #2: lockless-3.png --]
[-- Type: image/png, Size: 4089 bytes --]