Re: [PATCH 00/37] Permit filesystem local caching

[David Howells <dhowells@redhat.com>]

Daniel Phillips <phillips@phunq.net> wrote:

> This factor of four (even worse on XFS, not quite as bad on Ext3) is
> worth ruminating upon.  Is all of the difference explained by avoiding
> seeks on the server, which has the files in memory?

Here are some more stats for you to consider:

 (1) Copy the data across the network to a fresh Ext3 fs on the same partition
     I was using for the cache:

	[root@andromeda ~]# time cp -a /warthog/aaa /var/fscache
	real    0m39.052s
	user    0m0.368s
	sys     0m15.229s

 (2) Reboot and read back the files just written into Ext3 on the local disk:

	[root@andromeda ~]# time tar cf - /var/fscache/aaa >/dev/zero
	real    0m40.574s
	user    0m0.164s
	sys     0m3.512s

 (3) Run through the cache population process, and then run a tar directly on
     cachefiles's cache directly after a reboot:

	[root@andromeda ~]# time tar cf - /var/fscache/cache >/dev/zero
	real    4m53.104s
	user    0m0.192s
	sys     0m4.240s

So I guess there's a problem in cachefiles's efficiency - possibly due to the
fact that it tries to be fully asynchronous.

In case (1) this is very similar to the time for a read through a completely
cold cache (37.497s).

In case (2) this is comparable to cachefiles with a cache warmed prior to a
reboot (1m54.350s); in this case, however, cachefiles is doing some extra work:

 (a) It's doing a lookup on the server for each file, in addition to the
     lookups on the disk.  However, just doing a tar from plain NFS, the
     command completes in 22.330s.

 (b) It's reading an xattr per object for cache coherency management.

 (c) As the cache knows nothing of directories, files, etc., it lays its
     directory subtree out in a way that suits it.  File lookup keys are
     turned into filenames.  This may result in a less efficient arrangement
     in the cache than the original data, especially as directories may become
     very large, so Ext3 may be doing some extra work.

In case (3), this perhaps suggests that cachefiles's directory layout may be
part of the problem.  Running the following:

	ls -ldSr `find . -type d`

in /var/fscache/cache shows that the directories are either 4096 bytes in size
(158 instances) or 12288 bytes in size (105 instances), for a total of 263
directories.  There are 19255 files.

Running that ls command in /warthog/aaa shows 1185 directories, all but three
of them 4096 bytes in size; two are 12288 bytes and one is 20480 bytes in size
(include/linux/ unsurprisingly).  There are 19258 files, three of which are
hardlinks to other files in the tree.

> This could be easily tested by running a test against a server that is the
> same as the client, and does not have the files in memory.  If local access
> is still slower than network then there is a real issue with cache
> efficiency.

My server is also my desktop machine.  The only way to guarantee that the
memory is scrubbed is to reboot it:-(  I'll look at setting up one of my other
machines as an NFS server.

David