Re: [PATCH 00/45] Permit filesystem local caching [ver #41]

[David Howells <dhowells@redhat.com>]

Andrew Morton <akpm@linux-foundation.org> wrote:

> We would want to know the performance benefits in some detail before even 
> looking at the code, no?  Maybe they're in here somewhere but I missed it..

Okay...  You wanted some benchmarks, here are some.  I should try and automate
the procedure since it's pretty straightforward, just time consuming to do by
hand.


ENVIRONMENT
===========

I'm using a pair of computers, one an NFS server, the other an NFS client,
connected by ZyXEL PL-100 ethernet-over-mains adapters to throttle the network
bandwidth.

As far as I can tell, the TCP bandwidth as seen by a pair of netcats communing
with each other maxes out at about 890KB/s or 6.95 Mbits/s.  Amazon rates the
PL-100's as up to 85Mbits/s, but I don't seem to be getting anything like that.

The client was rebooted after each test, but the server wasn't.  The server was
persuaded to pull the entire working set for each test into RAM to eliminate
disk I/O latencies at that end.

The Ext3 partition used for the cache was tuned to have 4096-byte blocks.
During each run, a watch was put on the FS-Cache statistics on the client
machine:

	watch -n0 cat /proc/fs/fscache/stats

This went over SSH to my desktop machine by GigE ethernet.


FIRST BENCHMARK
===============

The first benchmark involved pulling a 100MB file by NFS to the client using
cat to /dev/zero run under time as a test.  The 'Time taken' reported by time
was logged.  The benchmark was repeated three times and the average was taken:

	Cache	RUN #1		RUN #2		RUN #3		AVG
	=======	===============	===============	===============	===============
	SERVER	0m0.062s
	NONE	1m59.462s	1m59.948s	2m1.852s	2.007 mins
	COLD	1m58.448s	1m59.436s	2m5.746s	2.020 mins
	HOT	0m2.235s	0m2.154s	0m2.171s	0.036 mins
	PGCACHE	0m0.040s

Firstly the test was run on the server twice and the second result logged
(SERVER).

Secondly, the client was rebooted and the test was run with the cachefilesd not
started and that was logged (NONE).  After rebooting, the cache contents were
erased (mke2fs) and cachefilesd was started and the test run again, which
loaded the cache (COLD).  Then the box was rebooted, cachefilesd was started
and the test run a third time, this time with a populated cache (HOT).  This
was repeated twice.

Finally, for reference, the client test was run again without unmounting,
stopping or rebooting anything so that the client's pagecache would act as the
cache (PGCACHE).


SECOND BENCHMARK
================

The second benchmark involved pulling a 256MB (as reported by du -s) kernel
tree or 1185 directories containing 19258 files using a single tar to /dev/zero
as a tesk.  The 'Time taken' reported by time was logged.  The benchmark was
repeated three times and the average was taken:

	Cache	RUN #1		RUN #2		RUN #3		AVG
	=======	===============	===============	===============	===============
	SERVER	0m0.348s
	NONE	7m35.335s	7m42.075s	7m32.797s	7.612 mins
	COLD	7m45.117s	7m54.774s	8m2.172s	7.900 mins
	HOT	7m14.970s	7m10.953s	7m16.390s	7.235 mins
	PGCACHE	3m10.864s

The procedure was as for the first benchmark.

For the second benchmark I also gathered data from the /proc/$$/mountstats file
to determine the network loading of run #3.  The following table shows the
counts of three different RPC operations issued, and the number of bytes read
over the network as part of READ RPC operations:

	Cache	GETATTR (N)	ACCESS (N)	READ (N)	READ (BYTES)
	=======	===============	===============	===============	===============
	NONE	22371		20486		21402		221252168
	COLD	22411		20486		21402		221252168
	HOT	22495		20481		0		0


CONCLUSION
==========

As can be seen, the network link I have between my test server and test client
is at about the break-even point for a large quantity of medium-small files (as
might be found in a source tree) with respect to the total time it takes to
completely read the files over NFS.

However, for those medium-small files, the reduction in network loading is huge
for repeat mass reads.  The time went from 7.6mins to 7.2mins, which is nice
but not hugely significant, but the network loading dropped by ~21,000 RPC
operations at a grand total of >220MB of data on the wire, allowing for network
metadata, within those 7 minutes.


For fewer but much larger files the cache has a proportionately greater effect
as the client incurs lower costs from Ext3 lookups as it is doing many fewer of
them, but gains greatly from Ext3's ability to glue large groups of contiguous
reads together and to do lookahead.  Similarly to the previous case, having
this data in the cache will reduce the network loading for repeat reads.


A comparison of the second benchmark test run against the server's pagecache
versus that test run against the client's pagecache is quite interesting.  The
server can perform the tar in a third of a second, but the client takes over
three minutes.

That would indicate that something on the order of just over 3 minutes's worth
of time is spent by each of the NONE, COLD and HOT test runs doing things other
than reads.  That would be GETATTR, ACCESS, and READDIRPLUS ops.

Another way of looking at it is that the NONE test or the second test spends a
little over 4 minutes doing READ ops from the network, and that the HOT test
spends almost as much time doing lookup, getxattr and read ops against Ext3.

It's also worth noting that the neither benchmark did the COLD test take very
much more time than the NONE test, despite doing lookups, mkdirs, creates,
setxattrs and writes in the background.


Of course, these two benchmarks are very much artificial: there was no other
significant loading on the network between the client and the server; there was
no other significant load on either machine; the cache started out empty and
probably got loaded in optimal order; the cache was large enough to never need
culling; only one program (cat or tar) was run at once.

David