Re: Lockless page cache test results

Nick Piggin
2006-04-27T06:04:59Z

Linus Torvalds wrote:

> The question, of course, is whether the part that remains (the actual page
> lookup) is important enough to matter, once it is part of a bigger chain
> in a real application.

It can be. If one applies a rough estimate, reading only 512MB/s
from the same file from pagecache on each CPU simply cannot scale
past about 16 CPUs before it becomes tree_lock bound (depending
on what sort of lock transfer latency numbers one plugs into the
model).

> And the "reading from the same file in multiple threads" _is_ a real load.
> It may sound stupid, but it would happen for any server that has a lot of
> locality across clients (and that's very much true for web-servers, for
> example).

I think something like postgresql, which uses a WAL via pagecache, things
that use shared memory (which is more important now that fork got lazier.

And it isn't even reading from the same file in multiple threads (although
that's obviously my poster child).

If one has F files in cache, and N CPUs each accessing a random file, the
number of files whos tree_lock a CPU last touched is F/N. The chance of a
CPU next hitting one of these is (F/N) / F = 1/N -> 0 pretty quickly.
include/ perhaps.

Now this won't be so bad as everyone piling up on a single cacheline, but
it will still hurt, especially on systems with fsb bus, broadcast snoop
(Opteron, POWER) etc.

Nor will it only be the read side which is significant. Consider a single
CPU reading a very large file, and each node's kswapd running on other CPUs
to reclaim pagecache (which needs to take the write lock).

>
> That said, under most real loads, the page cach elookup is obviously
> always going to be just a tiny tiny part (as shown by the fact that Jens
> quotes 35 GB/s throughput - possible only because splice to /dev/null
> doesn't need to actually ever even _touch_ the data).
>
> The fact that it drops to "just" 3GB/s for four clients is somewhat
> interesting, though, since that does put a limit on how well we can serve
> the same file (of course, 3GB/s is still a lot faster than any modern
> network will ever be able to push things around, but it's getting closer
> to the possibilities for real hardware (ie IB over PCI-X should be able to
> do about 1GB/s in "real life")
>
> So the fact that basically just lookup/locking overhead can limit things
> to 3GB/s is absolutely not totally uninteresting. Even if in practice
> there are other limits that would probably hit us much earlier.
>
> It would be interesting to see where doing gang-lookup moves the target,
> but on the other hand, with smaller files (and small files are still
> common), gang lookup isn't going to help as much.
>
> Of course, with small files, the actual filename lookup is likely to be
> the real limiter.

Although that's lockless so it scales. find_get_page will overtake it
at some point.

--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com