Re: [PATCH 00/37] Permit filesystem local caching

[David Howells <dhowells@redhat.com>]

Daniel Phillips <phillips@phunq.net> wrote:

> Have you got before/after benchmark results?

See attached.

These show a couple of things:

 (1) Dealing with lots of metadata slows things down a lot.  Note the result of
     looking and reading lots of small files with tar (the last result).  The
     NFS client has to both consult the NFS server *and* the cache.  Not only
     that, but any asynchronicity the cache may like to do is rendered
     ineffective by the fact tar wants to do a read on a file pretty much
     directly after opening it.

 (2) Getting metadata from the local disk fs is slower than pulling it across
     an unshared gigabit ethernet from a server that already has it in memory.

These points don't mean that fscache is no use, just that you have to consider
carefully whether it's of use to *you* given your particular situation, and
that depends on various factors.

Note that currently FS-Caching is disabled for individual NFS files opened for
writing as there's no way to handle the coherency problems thereby introduced.

David
---

			  ===========================
			  FS-CACHE FOR NFS BENCHMARKS
			  ===========================

 (*) The NFS client has a 1.86GHz Core2 Duo CPU and 1GB of RAM.

 (*) The NFS client has a Seagate ST380211AS 80GB 7200rpm SATA disk on an
     interface running in AHCI mode.  The chipset is an Intel G965.

 (*) A partition of approx 4.5GB is committed to caching, and is formatted as
     Ext3 with a blocksize of 4096 and directory indices.

 (*) The NFS client is using SELinux.

 (*) The NFS server is running an in-kernel NFSd, and has a 2.66GHz Core2 Duo
     CPU and 6GB of RAM.  The chipset is an Intel P965.

 (*) The NFS client is connected to the NFS server by Gigabit Ethernet.

 (*) The NFS mount is made with defaults for all options not relating to the
     cache:

	warthog:/warthog /warthog nfs
		rw,vers=3,rsize=1048576,wsize=1048576,hard,proto=tcp,timeo=600,
		retrans=2,sec=sys,fsc,addr=w.x.y.z 0 0


==================
FEW BIG FILES TEST
==================

Where:

 (*) The NFS server has two files:

	[root@andromeda ~]# ls -l /warthog/bigfile
	-rw-rw-r-- 1 4043 4043 104857600 2006-11-30 09:39 /warthog/bigfile
	[root@andromeda ~]# ls -l /warthog/biggerfile 
	-rw-rw-r-- 1 4043 4041 209715200 2006-03-21 13:56 /warthog/biggerfile

     Both of which are in memory on the server in all cases.


No patches, cold NFS cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.909s
	user    0m0.000s
	sys     0m0.520s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.750s
	user    0m0.000s
	sys     0m0.904s

CONFIG_FSCACHE=n, cold NFS cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.003s
	user    0m0.000s
	sys     0m0.124s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.100s
	user    0m0.004s
	sys     0m0.488s

Cold NFS cache, no disk cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.084s
	user    0m0.000s
	sys     0m0.136s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.020s
	user    0m0.000s
	sys     0m0.720s

Completely cold caches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.412s
	user    0m0.000s
	sys     0m0.892s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.449s
	user    0m0.000s
	sys     0m2.300s

Warm NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.067s
	user    0m0.000s
	sys     0m0.064s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.133s
	user    0m0.000s
	sys     0m0.136s

Warm Ext3 pagecache, cold NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.173s
	user    0m0.000s
	sys     0m0.172s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.316s
	user    0m0.000s
	sys     0m0.316s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.955s
	user    0m0.000s
	sys     0m0.244s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.596s
	user    0m0.000s
	sys     0m0.460s


===================================
MANY SMALL/MEDIUM FILE READING TEST
===================================

Where:

 (*) The NFS server has an old kernel tree:

	[root@andromeda ~]# du -s /warthog/aaa
	347340  /warthog/aaa
	[root@andromeda ~]# find /warthog/aaa | wc -l
	20443

     All of which is in memory on the server in all cases.


No patches, cold NFS cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m21.698s
	user    0m0.156s
	sys     0m5.284s

CONFIG_FSCACHE=n, cold NFS cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m22.337s
	user    0m0.152s
	sys     0m5.476s

Cold NFS cache, no disk cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m22.734s
	user    0m0.124s
	sys     0m5.796s

Completely cold caches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m37.497s
	user    0m0.248s
	sys     0m6.648s

Warm NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m15.167s
	user    0m0.168s
	sys     0m4.856s

Warm Ext3 pagecache, cold NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	tar: Removing leading `/' from member names
	tar: Removing leading `/' from hard link targets

	real    0m13.818s
	user    0m0.200s
	sys     0m5.492s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    1m54.350s
	user    0m0.044s
	sys     0m1.256s