Re: Lockless page cache test results

Jens Axboe
2006-04-27T07:44:58Z

[-- Attachment #1: Type: text/plain, Size: 2311 bytes --]

On Wed, Apr 26 2006, Andrew Morton wrote:
> Jens Axboe <axboe@suse.de> wrote:
> >
> > With a 16-page gang lookup in splice, the top profile for the 4-client
> > case (which is now at 4GiB/sec instead of 3) are:
> >
> > samples % symbol name
> > 30396 36.7217 __do_page_cache_readahead
> > 25843 31.2212 find_get_pages_contig
> > 9699 11.7174 default_idle
>
> __do_page_cache_readahead() should use gang lookup. We never got around to
> that, mainly because nothing really demonstrated a need.
>
> It's a problem that __do_page_cache_readahead() is being called at all -
> with everything in pagecache we should be auto-turning-off readahead. This
> happens because splice is calling the low-level do_pagecache_readahead().
> If you convert it to use page_cache_readahead(), that will all vanish if
> readahead is working right.

You are right, I modified it to use page_cache_readahead() and that
looks a lot better for the vanilla kernel. Here's a new graph of
2.6.17-rc3 and 2.6.17-rc3-lockless. Both base kernels are the splice
branch, so it contains the contig lookup change as well.

I'm still doing only up to nr_cpus clients, as numbers start to
fluctuate above that.

Things look pretty bad for the lockless kernel though, Nick any idea
what is going on there? The splice change is pretty simple, see the top
three patches here:

http://brick.kernel.dk/git/?p=linux-2.6-block.git;a=shortlog;h=splice

The top profile for the 4-client case looks like this:

samples % symbol name
77955 77.7141 find_get_pages_contig
8034 8.0092 splice_to_pipe
5092 5.0763 default_idle
4330 4.3166 do_splice_to
1323 1.3189 sys_splice

where vanilla is nowhere near that bad. I added a third graph, which is
lockless with the top patch backed out again so it's using
find_get_page() for each page. The top profile then looks like this for
the 4-client case:

samples % symbol name
10685 39.2730 default_idle
4120 15.1432 find_get_page
2608 9.5858 sys_splice
1729 6.3550 radix_tree_lookup_slot
1708 6.2778 splice_from_pipe
1071 3.9365 splice_to_pipe

Finally, I'll just note that the find_get_pages_contig() was modified
for the lockless kernel (read_lock -> spin_lock), so it isn't something
silly :-)

--
Jens Axboe

[-- Attachment #2: lockless-3.png --]
[-- Type: image/png, Size: 4089 bytes --]