RE: [PATCH 00/37] Permit filesystem local caching

RE: [PATCH 00/37] Permit filesystem local caching

[Rick Macklem]

> Well, the AFS paper that was referenced earlier was written around the
> time of 10bt and 100bt.  Local disk caching worked well then.  There
> should also be some papers at CITI about disk caching over slower
> connections, and disconnected operation (which should still be
> applicable today).  There are still winners from local disk caching, but
> their numbers have been reduced.  Server load reduction should be a win.
> I'm not sure if it's worth it from a security/manageability standpoint,
> but I haven't looked that closely at David's code.

One area that you might want to look at is WAN performance. When RPC RTT
goes up, ordinary NFS performance goes down. This tends to get overlooked
by the machine room folks. (There are several tools out there that can
introduce delay in an IP packet stream and emulate WAN RTTs.)

Just a thought, rick

[PATCH 00/37] Permit filesystem local caching

[David Howells]

These patches add local caching for network filesystems such as NFS.

The patches can roughly be broken down into a number of sets:

  (*) 01-keys-inc-payload.diff
  (*) 02-keys-search-keyring.diff
  (*) 03-keys-callout-blob.diff

      Three patches to the keyring code made to help the CIFS people.
      Included because of patches 05-08.

  (*) 04-keys-get-label.diff

      A patch to allow the security label of a key to be retrieved.
      Included because of patches 05-08.

  (*) 05-security-current-fsugid.diff
  (*) 06-security-separate-task-bits.diff
  (*) 07-security-subjective.diff
  (*) 08-security-kernel_service-class.diff
  (*) 09-security-kernel-service.diff
  (*) 10-security-nfsd.diff

      Patches to permit the subjective security of a task to be overridden.
      All the security details in task_struct are decanted into a new struct
      that task_struct then has two pointers two: one that defines the
      objective security of that task (how other tasks may affect it) and one
      that defines the subjective security (how it may affect other objects).

      Note that I have dropped the idea of struct cred for the moment.  With
      the amount of stuff that was excluded from it, it wasn't actually any
      use to me.  However, it can be added later.

      Required for cachefiles.

  (*) 11-release-page.diff
  (*) 12-fscache-page-flags.diff
  (*) 13-add_wait_queue_tail.diff
  (*) 14-fscache.diff

      Patches to provide a local caching facility for network filesystems.

  (*) 15-cachefiles-ia64.diff
  (*) 16-cachefiles-ext3-f_mapping.diff
  (*) 17-cachefiles-write.diff
  (*) 18-cachefiles-monitor.diff
  (*) 19-cachefiles-export.diff
  (*) 20-cachefiles.diff

      Patches to provide a local cache in a directory of an already mounted
      filesystem.

  (*) 21-nfs-comment.diff
  (*) 22-nfs-fscache-option.diff
  (*) 23-nfs-fscache-kconfig.diff
  (*) 24-nfs-fscache-top-index.diff
  (*) 25-nfs-fscache-server-obj.diff
  (*) 26-nfs-fscache-super-obj.diff
  (*) 27-nfs-fscache-inode-obj.diff
  (*) 28-nfs-fscache-use-inode.diff
  (*) 29-nfs-fscache-invalidate-pages.diff
  (*) 30-nfs-fscache-iostats.diff
  (*) 31-nfs-fscache-page-management.diff
  (*) 32-nfs-fscache-read-context.diff
  (*) 33-nfs-fscache-read-fallback.diff
  (*) 34-nfs-fscache-read-from-cache.diff
  (*) 35-nfs-fscache-store-to-cache.diff
  (*) 36-nfs-fscache-mount.diff
  (*) 37-nfs-fscache-display.diff

      Patches to provide NFS with local caching.

      A couple of questions on the NFS iostat changes: (1) Should I update the
      iostat version number; (2) is it permitted to have conditional iostats?


I've brought the patchset up to date with respect to the 2.6.25-rc1 merge
window, in particular altering Smack to handle the split in objective and
subjective security in the task_struct.

--
A tarball of the patches is available at:

	http://people.redhat.com/~dhowells/fscache/patches/nfs+fscache-30.tar.bz2


To use this version of CacheFiles, the cachefilesd-0.9 is also required.  It
is available as an SRPM:

	http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9-1.fc7.src.rpm

Or as individual bits:

	http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9.tar.bz2
	http://people.redhat.com/~dhowells/fscache/cachefilesd.fc
	http://people.redhat.com/~dhowells/fscache/cachefilesd.if
	http://people.redhat.com/~dhowells/fscache/cachefilesd.te
	http://people.redhat.com/~dhowells/fscache/cachefilesd.spec

The .fc, .if and .te files are for manipulating SELinux.

David

Re: [PATCH 00/37] Permit filesystem local caching

[Serge E. Hallyn]

Quoting David Howells (dhowells@redhat.com):
> 
> 
> These patches add local caching for network filesystems such as NFS.
> 
> The patches can roughly be broken down into a number of sets:
> 
>   (*) 01-keys-inc-payload.diff
>   (*) 02-keys-search-keyring.diff
>   (*) 03-keys-callout-blob.diff
> 
>       Three patches to the keyring code made to help the CIFS people.
>       Included because of patches 05-08.
> 
>   (*) 04-keys-get-label.diff
> 
>       A patch to allow the security label of a key to be retrieved.
>       Included because of patches 05-08.
> 
>   (*) 05-security-current-fsugid.diff
>   (*) 06-security-separate-task-bits.diff

Seems *really* weird that every time you send this, patch 6 doesn't seem
to reach me in any of my mailboxes...  (did get it from the url
you listed)

I'm sorry if I miss where you explicitly state this, but is it safe to
assume, as perusing the patches suggests, that

	1. tsk->sec never changes other than in task_alloc_security()?  

	2. tsk->act_as is only ever dereferenced from (a) current->
	   except (b) in do_coredump?

(thereby carefully avoiding locking issues)

I'd still like to see some performance numbers.  Not to object to
these patches, just to make sure there's no need to try and optimize
more of the dereferences away when they're not needed.

Oh, manually copied from patch 6, I see you have in the task_security
struct definition:

	kernel_cap_t    cap_bset;       /* ? */

That comment can be filled in with 'capability bounding set' (for this
task and all its future descendents).

thanks,
-serge

>   (*) 07-security-subjective.diff
>   (*) 08-security-kernel_service-class.diff
>   (*) 09-security-kernel-service.diff
>   (*) 10-security-nfsd.diff
> 
>       Patches to permit the subjective security of a task to be overridden.
>       All the security details in task_struct are decanted into a new struct
>       that task_struct then has two pointers two: one that defines the
>       objective security of that task (how other tasks may affect it) and one
>       that defines the subjective security (how it may affect other objects).
> 
>       Note that I have dropped the idea of struct cred for the moment.  With
>       the amount of stuff that was excluded from it, it wasn't actually any
>       use to me.  However, it can be added later.
> 
>       Required for cachefiles.
> 
>   (*) 11-release-page.diff
>   (*) 12-fscache-page-flags.diff
>   (*) 13-add_wait_queue_tail.diff
>   (*) 14-fscache.diff
> 
>       Patches to provide a local caching facility for network filesystems.
> 
>   (*) 15-cachefiles-ia64.diff
>   (*) 16-cachefiles-ext3-f_mapping.diff
>   (*) 17-cachefiles-write.diff
>   (*) 18-cachefiles-monitor.diff
>   (*) 19-cachefiles-export.diff
>   (*) 20-cachefiles.diff
> 
>       Patches to provide a local cache in a directory of an already mounted
>       filesystem.
> 
>   (*) 21-nfs-comment.diff
>   (*) 22-nfs-fscache-option.diff
>   (*) 23-nfs-fscache-kconfig.diff
>   (*) 24-nfs-fscache-top-index.diff
>   (*) 25-nfs-fscache-server-obj.diff
>   (*) 26-nfs-fscache-super-obj.diff
>   (*) 27-nfs-fscache-inode-obj.diff
>   (*) 28-nfs-fscache-use-inode.diff
>   (*) 29-nfs-fscache-invalidate-pages.diff
>   (*) 30-nfs-fscache-iostats.diff
>   (*) 31-nfs-fscache-page-management.diff
>   (*) 32-nfs-fscache-read-context.diff
>   (*) 33-nfs-fscache-read-fallback.diff
>   (*) 34-nfs-fscache-read-from-cache.diff
>   (*) 35-nfs-fscache-store-to-cache.diff
>   (*) 36-nfs-fscache-mount.diff
>   (*) 37-nfs-fscache-display.diff
> 
>       Patches to provide NFS with local caching.
> 
>       A couple of questions on the NFS iostat changes: (1) Should I update the
>       iostat version number; (2) is it permitted to have conditional iostats?
> 
> 
> I've brought the patchset up to date with respect to the 2.6.25-rc1 merge
> window, in particular altering Smack to handle the split in objective and
> subjective security in the task_struct.
> 
> --
> A tarball of the patches is available at:
> 
> 	http://people.redhat.com/~dhowells/fscache/patches/nfs+fscache-30.tar.bz2
> 
> 
> To use this version of CacheFiles, the cachefilesd-0.9 is also required.  It
> is available as an SRPM:
> 
> 	http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9-1.fc7.src.rpm
> 
> Or as individual bits:
> 
> 	http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9.tar.bz2
> 	http://people.redhat.com/~dhowells/fscache/cachefilesd.fc
> 	http://people.redhat.com/~dhowells/fscache/cachefilesd.if
> 	http://people.redhat.com/~dhowells/fscache/cachefilesd.te
> 	http://people.redhat.com/~dhowells/fscache/cachefilesd.spec
> 
> The .fc, .if and .te files are for manipulating SELinux.
> 
> David
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Serge E. Hallyn <serue@us.ibm.com> wrote:

> Seems *really* weird that every time you send this, patch 6 doesn't seem
> to reach me in any of my mailboxes...  (did get it from the url
> you listed)

It's the largest of the patches, so that's not entirely surprising.  Hence why
I included the URL to the tarball also.

> I'm sorry if I miss where you explicitly state this, but is it safe to
> assume, as perusing the patches suggests, that
> 
> 	1. tsk->sec never changes other than in task_alloc_security()?  

Correct.

> 	2. tsk->act_as is only ever dereferenced from (a) current->

That ought to be correct.

> 	   except (b) in do_coredump?

Actually, do_coredump() only deals with current->act_as.

> (thereby carefully avoiding locking issues)

That's the idea.

> I'd still like to see some performance numbers.  Not to object to
> these patches, just to make sure there's no need to try and optimize
> more of the dereferences away when they're not needed.

I hope that the performance impact is minimal.  The kernel should spend very
little time looking at the security data.  I'll try and get some though.

> Oh, manually copied from patch 6, I see you have in the task_security
> struct definition:
> 
> 	kernel_cap_t    cap_bset;       /* ? */
> 
> That comment can be filled in with 'capability bounding set' (for this
> task and all its future descendents).

Thanks.

David

Re: [PATCH 00/37] Permit filesystem local caching

[Daniel Phillips]

Hi David,

On Wednesday 20 February 2008 08:05, David Howells wrote:
> These patches add local caching for network filesystems such as NFS.

Have you got before/after benchmark results?

Regards,

Daniel

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> > These patches add local caching for network filesystems such as NFS.
> 
> Have you got before/after benchmark results?

I need to get a new hard drive for my test machine before I can go and get
some more up to date benchmark results.  It does seem, however, that the I/O
error handling capabilities of FS-Cache work properly:-)

David

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> Have you got before/after benchmark results?

See attached.

These show a couple of things:

 (1) Dealing with lots of metadata slows things down a lot.  Note the result of
     looking and reading lots of small files with tar (the last result).  The
     NFS client has to both consult the NFS server *and* the cache.  Not only
     that, but any asynchronicity the cache may like to do is rendered
     ineffective by the fact tar wants to do a read on a file pretty much
     directly after opening it.

 (2) Getting metadata from the local disk fs is slower than pulling it across
     an unshared gigabit ethernet from a server that already has it in memory.

These points don't mean that fscache is no use, just that you have to consider
carefully whether it's of use to *you* given your particular situation, and
that depends on various factors.

Note that currently FS-Caching is disabled for individual NFS files opened for
writing as there's no way to handle the coherency problems thereby introduced.

David
---

			  ===========================
			  FS-CACHE FOR NFS BENCHMARKS
			  ===========================

 (*) The NFS client has a 1.86GHz Core2 Duo CPU and 1GB of RAM.

 (*) The NFS client has a Seagate ST380211AS 80GB 7200rpm SATA disk on an
     interface running in AHCI mode.  The chipset is an Intel G965.

 (*) A partition of approx 4.5GB is committed to caching, and is formatted as
     Ext3 with a blocksize of 4096 and directory indices.

 (*) The NFS client is using SELinux.

 (*) The NFS server is running an in-kernel NFSd, and has a 2.66GHz Core2 Duo
     CPU and 6GB of RAM.  The chipset is an Intel P965.

 (*) The NFS client is connected to the NFS server by Gigabit Ethernet.

 (*) The NFS mount is made with defaults for all options not relating to the
     cache:

	warthog:/warthog /warthog nfs
		rw,vers=3,rsize=1048576,wsize=1048576,hard,proto=tcp,timeo=600,
		retrans=2,sec=sys,fsc,addr=w.x.y.z 0 0


==================
FEW BIG FILES TEST
==================

Where:

 (*) The NFS server has two files:

	[root@andromeda ~]# ls -l /warthog/bigfile
	-rw-rw-r-- 1 4043 4043 104857600 2006-11-30 09:39 /warthog/bigfile
	[root@andromeda ~]# ls -l /warthog/biggerfile 
	-rw-rw-r-- 1 4043 4041 209715200 2006-03-21 13:56 /warthog/biggerfile

     Both of which are in memory on the server in all cases.


No patches, cold NFS cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.909s
	user    0m0.000s
	sys     0m0.520s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.750s
	user    0m0.000s
	sys     0m0.904s

CONFIG_FSCACHE=n, cold NFS cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.003s
	user    0m0.000s
	sys     0m0.124s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.100s
	user    0m0.004s
	sys     0m0.488s

Cold NFS cache, no disk cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.084s
	user    0m0.000s
	sys     0m0.136s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.020s
	user    0m0.000s
	sys     0m0.720s

Completely cold caches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.412s
	user    0m0.000s
	sys     0m0.892s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.449s
	user    0m0.000s
	sys     0m2.300s

Warm NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.067s
	user    0m0.000s
	sys     0m0.064s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.133s
	user    0m0.000s
	sys     0m0.136s

Warm Ext3 pagecache, cold NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.173s
	user    0m0.000s
	sys     0m0.172s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.316s
	user    0m0.000s
	sys     0m0.316s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.955s
	user    0m0.000s
	sys     0m0.244s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.596s
	user    0m0.000s
	sys     0m0.460s


===================================
MANY SMALL/MEDIUM FILE READING TEST
===================================

Where:

 (*) The NFS server has an old kernel tree:

	[root@andromeda ~]# du -s /warthog/aaa
	347340  /warthog/aaa
	[root@andromeda ~]# find /warthog/aaa | wc -l
	20443

     All of which is in memory on the server in all cases.


No patches, cold NFS cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m21.698s
	user    0m0.156s
	sys     0m5.284s

CONFIG_FSCACHE=n, cold NFS cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m22.337s
	user    0m0.152s
	sys     0m5.476s

Cold NFS cache, no disk cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m22.734s
	user    0m0.124s
	sys     0m5.796s

Completely cold caches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m37.497s
	user    0m0.248s
	sys     0m6.648s

Warm NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m15.167s
	user    0m0.168s
	sys     0m4.856s

Warm Ext3 pagecache, cold NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	tar: Removing leading `/' from member names
	tar: Removing leading `/' from hard link targets

	real    0m13.818s
	user    0m0.200s
	sys     0m5.492s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    1m54.350s
	user    0m0.044s
	sys     0m1.256s

Re: [PATCH 00/37] Permit filesystem local caching

[Kevin Coffman]

On Thu, Feb 21, 2008 at 9:55 AM, David Howells <dhowells@redhat.com> wrote:
> Daniel Phillips <phillips@phunq.net> wrote:
>
>
> > Have you got before/after benchmark results?
>
>  See attached.
>
>  These show a couple of things:
>
>   (1) Dealing with lots of metadata slows things down a lot.  Note the result of
>      looking and reading lots of small files with tar (the last result).  The
>      NFS client has to both consult the NFS server *and* the cache.  Not only
>      that, but any asynchronicity the cache may like to do is rendered
>      ineffective by the fact tar wants to do a read on a file pretty much
>      directly after opening it.
>
>   (2) Getting metadata from the local disk fs is slower than pulling it across
>      an unshared gigabit ethernet from a server that already has it in memory.

Hi David,

Your results remind me of this in case you're interested...

http://www.citi.umich.edu/techreports/reports/citi-tr-92-3.pdf

Re: [PATCH 00/37] Permit filesystem local caching

[Daniel Phillips]

Hi David,

I am trying to spot the numbers that show the sweet spot for this 
optimization, without much success so far.

Who is supposed to win big?  Is this mainly about reducing the load on 
the server, or is the client supposed to win even with a lightly loaded 
server?

When you say Ext3 cache vs NFS cache is the first on the server and the 
second on the client?

Regards,

Daniel

RE: [PATCH 00/37] Permit filesystem local caching

[Muntz, Daniel]

Well, the AFS paper that was referenced earlier was written around the
time of 10bt and 100bt.  Local disk caching worked well then.  There
should also be some papers at CITI about disk caching over slower
connections, and disconnected operation (which should still be
applicable today).  There are still winners from local disk caching, but
their numbers have been reduced.  Server load reduction should be a win.
I'm not sure if it's worth it from a security/manageability standpoint,
but I haven't looked that closely at David's code.

  -Dan

-----Original Message-----
From: Daniel Phillips [mailto:phillips@phunq.net] 
Sent: Thursday, February 21, 2008 2:44 PM
To: David Howells
Cc: Myklebust, Trond; nfsv4@linux-nfs.org; linux-kernel@vger.kernel.org;
linux-fsdevel@vger.kernel.org; linux-security-module@vger.kernel.org;
selinux@tycho.nsa.gov; casey@schaufler-ca.com
Subject: Re: [PATCH 00/37] Permit filesystem local caching

Hi David,

I am trying to spot the numbers that show the sweet spot for this
optimization, without much success so far.

Who is supposed to win big?  Is this mainly about reducing the load on
the server, or is the client supposed to win even with a lightly loaded
server?

When you say Ext3 cache vs NFS cache is the first on the server and the
second on the client?

Regards,

Daniel
_______________________________________________
NFSv4 mailing list
NFSv4@linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> When you say Ext3 cache vs NFS cache is the first on the server and the 
> second on the client?

The filesystem on the server is pretty much irrelevant as long as (a) it
doesn't change, and (b) all the data is in memory on the server anyway.

The way the client works is like this:

	+---------+
	|         |                   
	|   NFS   |--+                
	|         |  |                
	+---------+  |   +----------+ 
	             |   |          | 
	+---------+  +-->|          | 
	|         |      |          |
	|   AFS   |----->| FS-Cache |
	|         |      |          |--+
	+---------+  +-->|          |  |
	             |   |          |  |   +--------------+   +--------------+
	+---------+  |   +----------+  |   |              |   |              |
	|         |  |                 +-->|  CacheFiles  |-->|  Ext3        |
	|  ISOFS  |--+                     |  /var/cache  |   |  /dev/sda6   |
	|         |                        +--------------+   +--------------+
	+---------+


 (1) NFS, say, asks FS-Cache to store/retrieve data for it;

 (2) FS-Cache asks the cache backend, in this case CacheFiles to honour the
     operation;

 (3) CacheFiles 'opens' a file in a mounted filesystem, say Ext3, and does read
     and write operations of a sort on it;

 (4) Ext3 decides how the cache data is laid out on disk - CacheFiles just
     attempts to use one sparse file per netfs inode.

> I am trying to spot the numbers that show the sweet spot for this 
> optimization, without much success so far.

What are you trying to do exactly?  Are you actually playing with it, or just
looking at the numbers I've produced?

> Who is supposed to win big?  Is this mainly about reducing the load on 
> the server, or is the client supposed to win even with a lightly loaded 
> server?

These are difficult questions to answer.  The obvious answer to both is "it
depends", and the real answer to both is "it's a compromise".

Inserting a cache adds overhead: you have to look in the cache to see if your
objects are mirrored there, and then you have to look in the cache to see if
the data you want is stored there; and then you might have to go to the server
anyway and then schedule a copy to be stored in the cache.

The characteristics of this type of cache depend on a number of things: the
filesystem backing it being the most obvious variable, but also how fragmented
it is and the properties of the disk drive or drives it is on.

Whether it's worth having a cache depend on the characteristics of the network
versus the characteristics of the cache.  Latency of the cache vs latency of
the network, for example.  Network loading is another: having a cache on each
of several clients sharing a server can reduce network traffic by avoiding the
read requests to the server.  NFS has a characteristic that it keeps spamming
the server with file status requests, so even if you take the read requests out
of the load, an NFS client still generates quite a lot of network traffic to
the server - but the reduction is still useful.

The metadata problem is quite a tricky one since it increases with the number
of files you're dealing with.  As things stand in my patches, when NFS, for
example, wants to access a new inode, it first has to go to the server to
lookup the NFS file handle, and only then can it go to the cache to find out if
there's a matching object in the case.  Worse, the cache must then perform
several synchronous disk bound metadata operations before it can be possible to
read from the cache.  Worse still, this means that a read on the network file
cannot proceed until (a) we've been to the server *plus* (b) we've been to the
disk.

The reason my client going to my server is so quick is that the server has the
dcache and the pagecache preloaded, so that across-network lookup operations
are really, really quick, as compared to the synchronous slogging of the local
disk to find the cache object.

I can probably improve this a little by pre-loading the subindex directories
(hash tables) that I use to reduce the directory size in the cache, but I don't
know by how much.


Anyway, to answer your questions:

 (1) It may help with heavily loaded networks with lots of read-only traffic.

 (2) It may help with slow connections (like doing NFS between the UK and
     Australia).

 (3) It could be used to do offline/disconnected operation.

David

Re: [PATCH 00/37] Permit filesystem local caching

[Daniel Phillips]

On Thursday 21 February 2008 16:07, David Howells wrote:
> The way the client works is like this:

Thanks for the excellent ascii art, that cleared up the confusion right
away.

> What are you trying to do exactly?  Are you actually playing with it, or just
> looking at the numbers I've produced?

Trying to see if you are offering enough of a win to justify testing it,
and if that works out, then going shopping for a bin of rotten vegetables
to throw at your design, which I hope you will perceive as useful.

In short I am looking for a reason to throw engineering effort at it.
>From the numbers you have posted I think you are missing some basic
efficiencies that could take this design from the sorta-ok zone to wow!

I think you may already be in the wow zone for taking load off a server
and I know of applications where an NFS server gets hammered so badly
that having the client suck a little in the unloaded case is a price
worth paying.  But the whole idea would be much more attractive if the
regressions were smaller.

> > Who is supposed to win big?  Is this mainly about reducing the load on 
> > the server, or is the client supposed to win even with a lightly loaded 
> > server?
> 
> These are difficult questions to answer.  The obvious answer to both is "it
> depends", and the real answer to both is "it's a compromise".
> 
> Inserting a cache adds overhead: you have to look in the cache to see if your
> objects are mirrored there, and then you have to look in the cache to see if
> the data you want is stored there; and then you might have to go to the server
> anyway and then schedule a copy to be stored in the cache.

But looking up the object in the cache should be nearly free - much less
than a microsecond per block.  If not then there are design issues.  I
suspect that you are doing yourself a disservice by going all the way
through the vfs to do this cache lookup, but this needs to be proved.

> The characteristics of this type of cache depend on a number of things: the
> filesystem backing it being the most obvious variable, but also how fragmented
> it is and the properties of the disk drive or drives it is on.

Double caching and vm unawareness of that has to hurt.

> The metadata problem is quite a tricky one since it increases with the number
> of files you're dealing with.  As things stand in my patches, when NFS, for
> example, wants to access a new inode, it first has to go to the server to
> lookup the NFS file handle, and only then can it go to the cache to find out if
> there's a matching object in the case.

So without the persistent cache it can omit the LOOKUP and just send the
filehandle as part of the READ?

> Worse, the cache must then perform 
> several synchronous disk bound metadata operations before it can be possible to
> read from the cache.  Worse still, this means that a read on the network file
> cannot proceed until (a) we've been to the server *plus* (b) we've been to the
> disk.
> 
> The reason my client going to my server is so quick is that the server has the 
> dcache and the pagecache preloaded, so that across-network lookup operations
> are really, really quick, as compared to the synchronous slogging of the local
> disk to find the cache object.

Doesn't that just mean you have to preload the lookup table for the
persistent cache so you can determine whether you are caching the data
for a filehandle without going to disk?

> I can probably improve this a little by pre-loading the subindex directories
> (hash tables) that I use to reduce the directory size in the cache, but I don't
> know by how much.

Ah I should have read ahead.  I think the correct answer is "a lot".
Your big can-t-get-there-from-here is the round trip to the server to
determine whether you should read from the local cache.  Got any ideas?

And where is the Trond-meister in all of this?

Regards,

Daniel

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> > The way the client works is like this:
> 
> Thanks for the excellent ascii art, that cleared up the confusion right
> away.

You know what they say about pictures... :-)

> > What are you trying to do exactly?  Are you actually playing with it, or
> > just looking at the numbers I've produced?
> 
> Trying to see if you are offering enough of a win to justify testing it,
> and if that works out, then going shopping for a bin of rotten vegetables
> to throw at your design, which I hope you will perceive as useful.

One thing that you have to remember: my test setup is pretty much the
worst-case for being appropriate for showing the need for caching to improve
performance.  There's a single client and a single server, they've got GigE
networking between them that has very little other load, and the server has
sufficient memory to hold the entire test data set.

> From the numbers you have posted I think you are missing some basic
> efficiencies that could take this design from the sorta-ok zone to wow!

Not really, it's just that this lashup could be considered designed to show
local caching in the worst light.

> But looking up the object in the cache should be nearly free - much less
> than a microsecond per block.

The problem is that you have to do a database lookup of some sort, possibly
involving several synchronous disk operations.

CacheFiles does a disk lookup by taking the key given to it by NFS, turning it
into a set of file or directory names, and doing a short pathwalk to the target
cache file.  Throwing in extra indices won't necessarily help.  What matters is
how quick the backing filesystem is at doing lookups.  As it turns out, Ext3 is
a fair bit better then BTRFS when the disk cache is cold.

> > The metadata problem is quite a tricky one since it increases with the
> > number of files you're dealing with.  As things stand in my patches, when
> > NFS, for example, wants to access a new inode, it first has to go to the
> > server to lookup the NFS file handle, and only then can it go to the cache
> > to find out if there's a matching object in the case.
> 
> So without the persistent cache it can omit the LOOKUP and just send the
> filehandle as part of the READ?

What 'it'?  Note that the get the filehandle, you have to do a LOOKUP op.  With
the cache, we could actually cache the results of lookups that we've done,
however, we don't know that the results are still valid without going to the
server:-/

AFS has a way around that - it versions its vnode (inode) IDs.

> > The reason my client going to my server is so quick is that the server has
> > the dcache and the pagecache preloaded, so that across-network lookup
> > operations are really, really quick, as compared to the synchronous
> > slogging of the local disk to find the cache object.
> 
> Doesn't that just mean you have to preload the lookup table for the
> persistent cache so you can determine whether you are caching the data
> for a filehandle without going to disk?

Where "lookup table" == "dcache".  That would be good yes.  cachefilesd
prescans all the files in the cache, which ought to do just that, but it
doesn't seem to be very effective.  I'm not sure why.

> > I can probably improve this a little by pre-loading the subindex
> > directories (hash tables) that I use to reduce the directory size in the
> > cache, but I don't know by how much.
> 
> Ah I should have read ahead.  I think the correct answer is "a lot".

Quite possibly.  It'll allow me to dispense with at least one fs lookup call
per cache object request call.

> Your big can-t-get-there-from-here is the round trip to the server to
> determine whether you should read from the local cache.  Got any ideas?

I'm not sure what you mean.  Your statement should probably read "... to
determine _what_ you should read from the local cache".

> And where is the Trond-meister in all of this?

Keeping quiet as far as I can tell.

David

Re: [PATCH 00/37] Permit filesystem local caching

[Daniel Phillips]

On Friday 22 February 2008 04:48, David Howells wrote:
> > But looking up the object in the cache should be nearly free - much less
> > than a microsecond per block.
> 
> The problem is that you have to do a database lookup of some sort, possibly
> involving several synchronous disk operations.

Right, so the obvious optimization strategy for this corner of it is to
decimate the synchronous disk ops for the average case, for which there
are a variety of options, one of which you already suggested.

> CacheFiles does a disk lookup by taking the key given to it by NFS, turning it
> into a set of file or directory names, and doing a short pathwalk to the target
> cache file.  Throwing in extra indices won't necessarily help.  What matters is
> how quick the backing filesystem is at doing lookups.  As it turns out, Ext3 is
> a fair bit better then BTRFS when the disk cache is cold.

All understood.  I am eventually going to suggest cutting the backing
filesystem entirely out of the picture, with a view to improving both
efficiency and transparency, hopefully with a code size reduction as
well.  But you are up and running with the filesystem approach, enough
to tackle the basic algorithm questions, which is worth a lot.

I really do not like idea of force fitting this cache into a generic
vfs model.  Sun was collectively smoking some serious crack when they
cooked that one up.  But there is also the ageless principle "isness is
more important than niceness".

> > > The metadata problem is quite a tricky one since it increases with the
> > > number of files you're dealing with.  As things stand in my patches, when
> > > NFS, for example, wants to access a new inode, it first has to go to the
> > > server to lookup the NFS file handle, and only then can it go to the cache
> > > to find out if there's a matching object in the case.
> > 
> > So without the persistent cache it can omit the LOOKUP and just send the
> > filehandle as part of the READ?
> 
> What 'it'?  Note that the get the filehandle, you have to do a LOOKUP op.  With
> the cache, we could actually cache the results of lookups that we've done,
> however, we don't know that the results are still valid without going to the
> server:-/

What I was trying to say.  It => the cache logic.

> AFS has a way around that - it versions its vnode (inode) IDs.

Which would require a change to NFS, not an option because you hope to
work with standard servers?  Of course with years to think about this,
the required protocol changes were put into v4.  Not.

/me hopes for an NFS hack to show up and explain the thinking there

Actually, there are many situations where changing both the client (you
must do that anyway) and the server is logistically practical.  In fact
that is true for all actual use cases I know of for this cache model.
So elaborating the protocol is not an option to reject out of hand.  A
hack along those lines could (should?) be provided as an opportunistic
option.

Have you completely exhausted optimization ideas for the file handle
lookup?

> > > The reason my client going to my server is so quick is that the server has
> > > the dcache and the pagecache preloaded, so that across-network lookup
> > > operations are really, really quick, as compared to the synchronous
> > > slogging of the local disk to find the cache object.
> > 
> > Doesn't that just mean you have to preload the lookup table for the
> > persistent cache so you can determine whether you are caching the data
> > for a filehandle without going to disk?
> 
> Where "lookup table" == "dcache".  That would be good yes.  cachefilesd
> prescans all the files in the cache, which ought to do just that, but it
> doesn't seem to be very effective.  I'm not sure why.

RCU?  Anyway, it is something to be tracked down and put right.

> > Your big can-t-get-there-from-here is the round trip to the server to
> > determine whether you should read from the local cache.  Got any ideas?
> 
> I'm not sure what you mean.  Your statement should probably read "... to
> determine _what_ you should read from the local cache".

What I tried to say.  So still... got any ideas?  That extra synchronous
network round trip is a killer.  Can it be made streaming/async to keep
throughput healthy?

> > And where is the Trond-meister in all of this?
> 
> Keeping quiet as far as I can tell.

/me does the Trond summoning dance

Daniel

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> I am eventually going to suggest cutting the backing filesystem entirely out
> of the picture,

You still need a database to manage the cache.  A filesystem such as Ext3
makes a very handy database for four reasons:

 (1) It exists and works.

 (2) It has a well defined interface within the kernel.

 (3) I can place my cache on, say, my root partition on my laptop.  I don't
     have to dedicate a partition to the cache.

 (4) Userspace cache management tools (such as cachefilesd) have an already
     existing interface to use: rmdir, unlink, open, getdents, etc..

I do have a cache-on-blockdev thing, but it's basically a wandering tree
filesystem inside.  It is, or was, much faster than ext3 on a clean cache, but
it degrades horribly over time because my free space reclamation sucks - it
gradually randomises the block allocation sequence over time.

So, what would you suggest instead of a backing filesystem?

> I really do not like idea of force fitting this cache into a generic
> vfs model.  Sun was collectively smoking some serious crack when they
> cooked that one up.  But there is also the ageless principle "isness is
> more important than niceness".

What do you mean?  I'm not doing it like Sun.  The cache is a side path from
the netfs.  It should be transparent to the user, the VFS and the server.

The only place it might not be transparent is that you might to have to
instruct the netfs mount to use the cache.  I'd prefer to do it some other way
than passing parameters to mount, though, as (1) this causes fun with NIS
distributed automounter maps, and (2) people are asking for a finer grain of
control than per-mountpoint.  Unfortunately, I can't seem to find a way to do
it that's acceptable to Al.

> Which would require a change to NFS, not an option because you hope to
> work with standard servers?  Of course with years to think about this,
> the required protocol changes were put into v4.  Not.

I don't think there's much I can do about NFS.  It requires the filesystem
from which the NFS server is dealing to have inode uniquifiers, which are then
incorporated into the file handle.  I don't think the NFS protocol itself
needs to change to support this.

> Have you completely exhausted optimization ideas for the file handle
> lookup?

No, but there aren't many.  CacheFiles doesn't actually do very much, and it's
hard to reduce that not very much.  The most obvious thing is to prepopulate
the dcache, but that's at the expense of memory usage.

Actually, if I cache the name => FH mapping I used last time, I can make a
start on looking up in the cache whilst simultaneously accessing the server.
If what's on the server has changed, I can ditch the speculative cache lookup
I was making and start a new cache lookup.

However, storing directory entries has penalties of its own, though it'll be
necesary if we want to do disconnected operation.

> > Where "lookup table" == "dcache".  That would be good yes.  cachefilesd
> > prescans all the files in the cache, which ought to do just that, but it
> > doesn't seem to be very effective.  I'm not sure why.
> 
> RCU?  Anyway, it is something to be tracked down and put right.

cachefilesd runs in userspace.  It's possible it isn't doing enough to preload
all the metadata.

> What I tried to say.  So still... got any ideas?  That extra synchronous
> network round trip is a killer.  Can it be made streaming/async to keep
> throughput healthy?

That's a per-netfs thing.  With the test rig I've got, it's going to the
on-disk cache that's the killer.  Going over the network is much faster.

See the results I posted.  For the tarball load, and using Ext3 to back the
cache:

	Cold NFS cache, no disk cache:		0m22.734s
	Warm on-disk cache, cold pagecaches:	1m54.350s

The problem is reading using tar is a worst case workload for this.  Everything
it does is pretty much completely synchronous.

One thing that might help is if things like tar and find can be made to use
fadvise() on directories to hint to the filesystem (NFS, AFS, whatever) that
it's going to access every file in those directories.

Certainly AFS could make use of that: the directory is read as a file, and the
netfs then parses the file to get a list of vnode IDs that that directory
points to.  It could then do bulk status fetch operations to instantiate the
inodes 50 at a time.

I don't know whether NFS could use it.  Someone like Trond or SteveD or Chuck
would have to answer that.

David

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

David Howells <dhowells@redhat.com> wrote:

> > Have you got before/after benchmark results?
> 
> See attached.

Attached here are results using BTRFS (patched so that it'll work at all)
rather than Ext3 on the client on the partition backing the cache.

Note that I didn't bother redoing the tests that didn't involve a cache as the
choice of filesystem backing the cache should have no bearing on the result.

Generally, completely cold caches shouldn't show much variation as all the
writing can be done completely asynchronously, provided the client doesn't
fill its RAM.

The interesting case is where the disk cache is warm, but the pagecache is
cold (ie: just after a reboot after filling the caches).  Here, for the two
big files case, BTRFS appears quite a bit better than Ext3, showing a 21%
reduction in time for the smaller case and a 13% reduction for the larger
case.

For the many small/medium files case, BTRFS performed significantly better
(15% reduction in time) in the case where the caches were completely cold.
I'm not sure why, though - perhaps because it doesn't execute a write_begin()
stage during the write_one_page() call and thus doesn't go allocating disk
blocks to back the data, but instead allocates them later.

More surprising is that BTRFS performed significantly worse (15% increase in
time) in the case where the cache on disk was fully populated and then the
machine had been rebooted to clear the pagecaches.

It's important to note that I've only run each test once apiece, so the
numbers should be taken with a modicum of salt (bad statistics and all that).

David
---
===========================
FEW BIG FILES TEST ON BTRFS
===========================

Completely cold caches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.124s
	user    0m0.000s
	sys     0m1.260s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.538s
	user    0m0.000s
	sys     0m2.624s

Warm NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.061s
	user    0m0.000s
	sys     0m0.064s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.118s
	user    0m0.000s
	sys     0m0.116s

Warm BTRFS pagecache, cold NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.189s
	user    0m0.000s
	sys     0m0.188s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.369s
	user    0m0.000s
	sys     0m0.368s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.540s
	user    0m0.000s
	sys     0m1.440s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.132s
	user    0m0.000s
	sys     0m1.724s


============================================
MANY SMALL/MEDIUM FILE READING TEST ON BTRFS
============================================

Completely cold caches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m31.838s
	user    0m0.192s
	sys     0m6.076s

Warm NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m14.841s
	user    0m0.148s
	sys     0m4.988s

Warm BTRFS pagecache, cold NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m16.773s
	user    0m0.148s
	sys     0m5.512s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    2m12.527s
	user    0m0.080s
	sys     0m2.908s

Re: [PATCH 00/37] Permit filesystem local caching

[Chris Mason]

On Thursday 21 February 2008, David Howells wrote:
> David Howells <dhowells@redhat.com> wrote:
> > > Have you got before/after benchmark results?
> >
> > See attached.
>
> Attached here are results using BTRFS (patched so that it'll work at all)
> rather than Ext3 on the client on the partition backing the cache.

Thanks for trying this, of course I'll ask you to try again with the latest 
v0.13 code, it has a number of optimizations especially for CPU usage.

>
> Note that I didn't bother redoing the tests that didn't involve a cache as
> the choice of filesystem backing the cache should have no bearing on the
> result.
>
> Generally, completely cold caches shouldn't show much variation as all the
> writing can be done completely asynchronously, provided the client doesn't
> fill its RAM.
>
> The interesting case is where the disk cache is warm, but the pagecache is
> cold (ie: just after a reboot after filling the caches).  Here, for the two
> big files case, BTRFS appears quite a bit better than Ext3, showing a 21%
> reduction in time for the smaller case and a 13% reduction for the larger
> case.

I'm afraid I don't have a good handle on the filesystem operations that result 
from this workload.  Are we reading from the FS to fill the NFS page cache?

>
> For the many small/medium files case, BTRFS performed significantly better
> (15% reduction in time) in the case where the caches were completely cold.
> I'm not sure why, though - perhaps because it doesn't execute a
> write_begin() stage during the write_one_page() call and thus doesn't go
> allocating disk blocks to back the data, but instead allocates them later.

If your write_one_page call does parts of btrfs_file_write, you'll get delayed 
allocation for anything bigger than 8k by default.  <= 8k will get packed 
into the btree leaves.

>
> More surprising is that BTRFS performed significantly worse (15% increase
> in time) in the case where the cache on disk was fully populated and then
> the machine had been rebooted to clear the pagecaches.

Which FS operations are included here?  Finding all the files or just an 
unmount?  Btrfs defrags metadata in the background, and unmount has to wait 
for that defrag to finish.

Thanks again,
Chris

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Chris Mason <chris.mason@oracle.com> wrote:

> > The interesting case is where the disk cache is warm, but the pagecache is
> > cold (ie: just after a reboot after filling the caches).  Here, for the two
> > big files case, BTRFS appears quite a bit better than Ext3, showing a 21%
> > reduction in time for the smaller case and a 13% reduction for the larger
> > case.
> 
> I'm afraid I don't have a good handle on the filesystem operations that
> result from this workload.  Are we reading from the FS to fill the NFS page
> cache?

I'm not sure what you're asking.

When the cache is cold, we determine that we can't read from the cache very
quickly.  We then read data from the server and, in the background, create the
metadata in the cache and store the data to it (by copying netfs pages to
backingfs pages).

When the cache is warm, we read the data from the cache, copying the data from
the backingfs pages to the netfs pages.  We use bmap() to ascertain that there
is data to be read, otherwise we detect a hole and fallback to reading from
the server.

Looking up cache object involves a sequence of lookup() ops and getxattr() ops
on the backingfs.  Should an object not exist, we defer creation of that
object to a background thread and do lookups(), mkdirs() and setxattrs() and a
create() to manufacture the object.

We read data from an object by calling readpages() on the backingfs to bring
the data into the pagecache.  We monitor the PG_lock bits to find out when
each page is read or has completed with an error.

Writing pages to the cache is done completely in the background.
PG_fscache_write is set on a page when it is handed to fscache to storage,
then at some point a background thread wakes up and calls write_one_page() in
the backingfs to write that page to the cache file.  At the moment, this
copies the data into a backingfs page which is then marked PG_dirty, and the
VM writes it out in the usual way.

> > More surprising is that BTRFS performed significantly worse (15% increase
> > in time) in the case where the cache on disk was fully populated and then
> > the machine had been rebooted to clear the pagecaches.
> 
> Which FS operations are included here?  Finding all the files or just an 
> unmount?  Btrfs defrags metadata in the background, and unmount has to wait 
> for that defrag to finish.

BTRFS might not be doing any writing at all here - apart from local atimes
(used by cache culling), that is.

What it does have to do is lots of lookups, reads and getxattrs, all of which
are synchronous.

David

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Chris Mason <chris.mason@oracle.com> wrote:

> Thanks for trying this, of course I'll ask you to try again with the latest 
> v0.13 code, it has a number of optimizations especially for CPU usage.

Here you go.  The numbers are very similar.

David

=================================
FEW BIG FILES TEST ON BTRFS v0.13
=================================

Completely cold caches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.202s
	user    0m0.000s
	sys     0m1.716s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.212s
	user    0m0.000s
	sys     0m0.896s

Warm BTRFS pagecache, cold NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.197s
	user    0m0.000s
	sys     0m0.192s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.376s
	user    0m0.000s
	sys     0m0.372s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.543s
	user    0m0.004s
	sys     0m1.448s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.111s
	user    0m0.000s
	sys     0m2.856s


==================================================
MANY SMALL/MEDIUM FILE READING TEST ON BTRFS v0.13
==================================================

Completely cold caches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m31.575s
	user    0m0.176s
	sys     0m6.316s

Warm BTRFS pagecache, cold NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m16.081s
	user    0m0.164s
	sys     0m5.528s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    2m15.245s
	user    0m0.064s
	sys     0m2.808s

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> This factor of four (even worse on XFS, not quite as bad on Ext3) is
> worth ruminating upon.  Is all of the difference explained by avoiding
> seeks on the server, which has the files in memory?

Here are some more stats for you to consider:

 (1) Copy the data across the network to a fresh Ext3 fs on the same partition
     I was using for the cache:

	[root@andromeda ~]# time cp -a /warthog/aaa /var/fscache
	real    0m39.052s
	user    0m0.368s
	sys     0m15.229s

 (2) Reboot and read back the files just written into Ext3 on the local disk:

	[root@andromeda ~]# time tar cf - /var/fscache/aaa >/dev/zero
	real    0m40.574s
	user    0m0.164s
	sys     0m3.512s

 (3) Run through the cache population process, and then run a tar directly on
     cachefiles's cache directly after a reboot:

	[root@andromeda ~]# time tar cf - /var/fscache/cache >/dev/zero
	real    4m53.104s
	user    0m0.192s
	sys     0m4.240s

So I guess there's a problem in cachefiles's efficiency - possibly due to the
fact that it tries to be fully asynchronous.

In case (1) this is very similar to the time for a read through a completely
cold cache (37.497s).

In case (2) this is comparable to cachefiles with a cache warmed prior to a
reboot (1m54.350s); in this case, however, cachefiles is doing some extra work:

 (a) It's doing a lookup on the server for each file, in addition to the
     lookups on the disk.  However, just doing a tar from plain NFS, the
     command completes in 22.330s.

 (b) It's reading an xattr per object for cache coherency management.

 (c) As the cache knows nothing of directories, files, etc., it lays its
     directory subtree out in a way that suits it.  File lookup keys are
     turned into filenames.  This may result in a less efficient arrangement
     in the cache than the original data, especially as directories may become
     very large, so Ext3 may be doing some extra work.

In case (3), this perhaps suggests that cachefiles's directory layout may be
part of the problem.  Running the following:

	ls -ldSr `find . -type d`

in /var/fscache/cache shows that the directories are either 4096 bytes in size
(158 instances) or 12288 bytes in size (105 instances), for a total of 263
directories.  There are 19255 files.

Running that ls command in /warthog/aaa shows 1185 directories, all but three
of them 4096 bytes in size; two are 12288 bytes and one is 20480 bytes in size
(include/linux/ unsurprisingly).  There are 19258 files, three of which are
hardlinks to other files in the tree.

> This could be easily tested by running a test against a server that is the
> same as the client, and does not have the files in memory.  If local access
> is still slower than network then there is a real issue with cache
> efficiency.

My server is also my desktop machine.  The only way to guarantee that the
memory is scrubbed is to reboot it:-(  I'll look at setting up one of my other
machines as an NFS server.

David

Re: [PATCH 00/37] Permit filesystem local caching

[Daniel Phillips]

On Monday 25 February 2008 15:19, David Howells wrote:
> So I guess there's a problem in cachefiles's efficiency - possibly due
> to the fact that it tries to be fully asynchronous.

OK, not just my imagination, and it makes me feel better about the patch 
set because efficiency bugs are fixable while fundamental limitations 
are not.

How much of a hurry are you in to merge this feature?  You have bits 
like this:

"Add a function to install a monitor on the page lock waitqueue for a 
particular page, thus allowing the page being unlocked to be detected.
This is used by CacheFiles to detect read completion on a page in the 
backing filesystem so that it can then copy the data to the waiting 
netfs page."

We already have that hook, it is called bio_endio.  My strong intuition 
is that your whole mechanism should sit directly on the block device, 
no matter how attractive it seems to be able to piggyback on the 
namespace and layout management code of existing filesystems.  I see 
your current effort as the moral equivalent of FUSE: you are able to 
demonstrate certain desirable behavioral properties, but you are unable 
to reach full theoretical efficiency because there are layers and 
layers of interface gunk interposed between the netfs user and the 
cache device.

That said, I also see you have put a huge amount of work into this over 
the years, it is nicely broken out, you are responsive and easy to work 
with, all arguments for an early merge.  Against that, you invade core 
kernel for reasons that are not necessarily justified:

  * two new page flags
  * a new fileops method
  * many changes to LSM including new object class and new hooks
  * separate fs*id from task struct
  * new page-private destructor hook
  * probably other bits I missed

Would it be correct to say that some of these changes are to support 
disconnected operation?  If so, you really have two patch sets:

  1) Persistent netfs cache
  2) Disconnected netfs operation

You have some short snappers that look generally useful:

  * add_wait_queue_tail (cool)
  * write to a file without a struct file (includes ->mapping cleanup,
    probably good)
  * export fsync_super

Why not hunt around for existing in-kernel users that would benefit so 
these can be submitted as standalone patches, shortening the remaining 
patch set and partially overcoming objections due to core kernel 
changes?

One thing I don't see is users coming on to lkml and saying "please 
merge this, it works great for me".  Since you probably have such 
users, why not give them a poke? 

Your cachefilesd is going to need anti-deadlock medicine like ddsnap 
has.  Since you don't seem at all worried about that right now, I 
suspect you have not hammered this code really heavily, correct?  
Without preventative measures, any memory-using daemon sitting in the 
block IO path will deadlock if you hit it hard enough.

A couple of years ago you explained the purpose of the new page flags to 
me and there is no way I can find that email again.  Could you explain 
it again please?  Meanwhile I am doing my duty and reading your OLS 
slides etc.

Regards,

Daniel

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> On Monday 25 February 2008 15:19, David Howells wrote:
> > So I guess there's a problem in cachefiles's efficiency - possibly due
> > to the fact that it tries to be fully asynchronous.
> 
> OK, not just my imagination, and it makes me feel better about the patch 
> set because efficiency bugs are fixable while fundamental limitations 
> are not.

One can hope:-)

> How much of a hurry are you in to merge this feature?  You have bits 
> like this:

I'd like to get it upstream sooner rather than later.  As it's not upstream,
but it's prerequisite patches touch a lot of code, I have to spend time
regularly making my patches work again.  Merge windows are completely not fun.

> "Add a function to install a monitor on the page lock waitqueue for a 
> particular page, thus allowing the page being unlocked to be detected.
> This is used by CacheFiles to detect read completion on a page in the 
> backing filesystem so that it can then copy the data to the waiting 
> netfs page."
> 
> We already have that hook, it is called bio_endio.

Except that isn't accessible.  CacheFiles currently has no access to the
notification from the blockdev to the backing fs, if indeed there is one.  All
we can do it trap the backing fs page becoming available.

> My strong intuition is that your whole mechanism should sit directly on the
> block device, no matter how attractive it seems to be able to piggyback on
> the namespace and layout management code of existing filesystems.

There's a place for both.

Consider a laptop with a small disk, possibly subdivided between Linux and
Windows.  Linux then subdivides its bit further to get a swap space.  What you
then propose is to break off yet another chunk to provide the cache.  You
can't then use this other chunk for anything else, even if it's, say, 1% used
by the cache.

The way CacheFiles works is that you tell it that it can use up to a certain
percentage of the otherwise free disk space on an otherwise existing
filesystem.  In the laptop case, you may just have a single big partition.  The
cache will fill up as much of it can, and as the other contents of the
partition consume space, the cache will be culled to make room.

On the other hand, a system like my desktop, where I can slap in extra disks
with mound of extra disk space, it might very well make sense to commit block
devices to caching, as this can be used to gain performance.

I have another cache backend (CacheFS) which takes the form of a filesystem,
thus allowing you to mount a blockdev as a cache.  It's much faster than Ext3
at storing and retrieving files... at first.  The problem is that I've mucked
up the free space retrieval such that performance degrades by 20x over time for
files of any size.

Basically any cache on a raw blockdev _is_ a filesystem, just one in which
you're randomly allowed to discard data to make life easier.

> I see your current effort as the moral equivalent of FUSE: you are able to
> demonstrate certain desirable behavioral properties, but you are unable to
> reach full theoretical efficiency because there are layers and layers of
> interface gunk interposed between the netfs user and the cache device.

The interface gunk is meant to be as thin as possible, but there are
constraints (see the documentation in the main FS-Cache patch for more
details):

 (1) It's a requirement that it only be tied to, say, AFS.  We might have
     several netfs's that want caching: AFS, CIFS, ISOFS (okay, that last isn't
     really a netfs, but it might still want caching).

 (2) I want to be able to change the backing cache.  Under some circumstances I
     might want to use an existing filesystem, under others I might want to
     commit a blockdev.  I've even been asked about using battery-backed RAM -
     which has different design constraints.

 (3) The constraint has been imposed by the NFS team that the cache be
     completely asynchronous.  I haven't quite met this: readpages() will wait
     until the cache knows whether or not the pages are available on the
     principle that read operations done through the cache can be considered
     synchronous.  This is an attempt to reduce the context switchage involved.

Unfortunately, the asynchronicity requirement has caused the middle layer to
bloat.  Fortunately, the backing cache needn't bloat as it can use the middle
layer's bloat.

> That said, I also see you have put a huge amount of work into this over 
> the years, it is nicely broken out, you are responsive and easy to work 
> with, all arguments for an early merge.  Against that, you invade core 
> kernel for reasons that are not necessarily justified:
>
>   * two new page flags

I need to keep track of two bits of per-cached-page information:

 (1) This page is known by the cache, and that the cache must be informed if
     the page is going to go away.

 (2) This page is being written to disk by the cache, and that it cannot be
     released until completion.  Ideally it shouldn't be changed until
     completion either so as to maintain the known state of the cache.

I could set up a radix tree per data storage object to keep track of both these
points, however this would mean that the netfs would have to do a call,
spinlock, conditional jumps, etc to find out either state.

On the other hand, if we can spare two page flags, those are sufficient.

Note that the cache doesn't necessarily need to be able to find the netfs
pages, but may have to pin resources for backing them.

Further note that PG_private may not be used as I want to be able to use
caching with ISOFS eventually.

>   * a new fileops method

Do you mean a new address space ops method?  Yes.  I have to be able to write
from a kernel page without the use of a struct file.  The struct file isn't
actually necessary to do the write, and so is a waste of space.  What's worse
is that the struct file plays havoc with resource limits and ENFILE production.

Ideally I want a couple of hooks: one to do O_DIRECT writing to a file from
kernel pages, one to do O_DIRECT|O_NOHOLE reading from a file to kernel pages
(holes in cache files represent blocks not retrieved from the server, so I want
to see ENODATA not a block of zeros).

>   * many changes to LSM including new object class and new hooks
>   * separate fs*id from task struct

It has been required that I call vfs_mkdir() and suchlike rather than bypassing
security and calling inode ops directly.  Therefore the VFS and LSM get to deny
the caching kernel modules access to the cache data because under some
circumstances the caching code is running in the security context of whatever
process issued the original syscall on the netfs.

Furthermore, the security parameters with which a file is created (UID, GID,
security label) would be derived from that process that issued the system call,
thus potentially preventing other processes from accessing the cache, in
particular cachefilesd.

So, what is required is to temporarily override the security of the process
that issued the system call.  We can't, however, just do an in-place change of
the security data as that affects the process as an object, not just as a
subject.  This means it may lose signals or ptrace events for example, and
affect what the task looks like in /proc.

So what I've done is to make a logical slit in the security between the
objective security (task->sec) and the subjective security (task->act_as).  The
objective security holds the intrinsic security properties of a process and is
never overridden.  This is what appears in /proc, and is used when a process is
the target of an operation by some other process (SIGKILL for example).

The subjective security holds the active security properties of a process, and
may be overridden.  This is not seen externally, and is used whan a process
acts upon another object, for example SIGKILLing another process or opening a
file.

The new hooks allow SELinux (or Smack or whatever) to reject a request for a
kernel service (such as cachefiles) to run in a context of a specific security
label or to create files and directories with another security label.  These
hooks may also be useful for NFSd.

>   * new page-private destructor hook

The cache may attach state to pages before read_cache_pages() is called.
Therefore read_cache_pages() may need to arrange for it to be cleaned up.  The
only way it can know to do this is by examining the page flags.  PG_private may
not be overloaded because it is owned by fs/buffer.c and friends on things like
ISOFS.

>   * probably other bits I missed

Note that most of these things have been muchly argued over already.

> Would it be correct to say that some of these changes are to support 
> disconnected operation?

No.

> You have some short snappers that look generally useful:
> 
>   * add_wait_queue_tail (cool)

Which you complained about above.

>   * write to a file without a struct file (includes ->mapping cleanup,
>     probably good)

Ditto.

>   * export fsync_super

> Why not hunt around for existing in-kernel users that would benefit so 
> these can be submitted as standalone patches, shortening the remaining 
> patch set and partially overcoming objections due to core kernel 
> changes?

The only ones that really fall into that category are the security patches,
which admittedly affect a lot of places.  That might be acceptable, and the
thought has occurred to me, because of NFSd.

> One thing I don't see is users coming on to lkml and saying "please 
> merge this, it works great for me".  Since you probably have such 
> users, why not give them a poke? 

The problem is that I'm stuck on waiting for the NFS guys to okay the NFS
patches.

> Your cachefilesd is going to need anti-deadlock medicine like ddsnap 
> has.

You mean the userspace daemon?  Why should it deadlock?

> Since you don't seem at all worried about that right now, I suspect you have
> not hammered this code really heavily, correct?

I had run iozone on cached NFS prior to asynchronising it.  However, I've found
a bug in my thread pool code that I'm currently chasing, so I need to do more
parallelisation testing

> Without preventative measures, any memory-using daemon sitting in the block
> IO path will deadlock if you hit it hard enough.

cachefilesd doesn't actually seem to consume that much memory, and it's
unlikely to deadlock as it only does one thing at once and has no locking.

There is a potential race though between cachefilesd's cull scanner and someone
scanning through all the files that are cached in the same order over and over
again.  The problem is that we cannot keep the view of old stuff in the cache
up to date, no matter how hard we try.  I haven't thought of a good way around
that.

> A couple of years ago you explained the purpose of the new page flags to 
> me and there is no way I can find that email again.  Could you explain 
> it again please?  Meanwhile I am doing my duty and reading your OLS 
> slides etc.

See above.

David

Re: [PATCH 00/37] Permit filesystem local caching

[Daniel Phillips]

I need to respond to this in pieces... first the bit that is bugging
me:

> >   * two new page flags
> 
> I need to keep track of two bits of per-cached-page information:
> 
>  (1) This page is known by the cache, and that the cache must be informed if
>      the page is going to go away.

I still do not understand the life cycle of this bit.  What does the
cache do when it learns the page has gone away?  How is it informed?
Who owns the page cache in which such a page lives, the nfs client?
Filesystem that hosts the page?  A third page cache owned by the
cache itself?  (See my basic confusion about how many page cache
levels you have, below.)

Suppose one were to take a mundane approach to the persistent cache
problem instead of layering filesystems.  What you would do then is
change NFS's ->write_page and variants to fiddle the persistent
cache as well as the network, instead of just the network as now.
This fiddling could even consist of ->write calls to another
filesystem, though working directly with the bio interface would
yield the fastest, and therefore to my mind, best result.

In any case, you find out how to write the page to backing store by
asking the filesystem, which in the naive approach would be nfs
augmented with caching library calls.  The filesystem keeps its own
metadata around to know how to map the page to disk.  So again
naively, this metadata could tell the nfs client that the page is
not mapped to disk at all.  So I do not see what your per-page bit
is for, obviously because I do not fully understand your caching
scheme.  Which I could eventually find out by reading all the
patches but asking you is so much more fun :-)

By the way, how many levels of page caching for the same data are
there, is it:

  1) nfs client
  2) cache layer's own page cache
  3) filesystem hosting the cache

or just:

  1) nfs client page cache
  2) filesystem hosting the cache

I think it is the second, but that is already double caching, which
has got to hurt.

Regards,

Daniel

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> I need to respond to this in pieces... first the bit that is bugging
> me:
> 
> > >   * two new page flags
> > 
> > I need to keep track of two bits of per-cached-page information:
> > 
> >  (1) This page is known by the cache, and that the cache must be informed if
> >      the page is going to go away.
> 
> I still do not understand the life cycle of this bit.  What does the
> cache do when it learns the page has gone away?

That's up to the cache.  CacheFS, for example, unpins some resources when all
the pages managed by a pointer block are taken away from it.  The cache may
also reserve a block on disk to back this page, and that reservation may then
be discarded by the netfs uncaching the page.

The cache may also speculatively take copies of the page if the machine is
idle.

Documentation/filesystems/caching/netfs-api.txt describes the caching API as a
process, including the presentation of netfs pages to the cache and their
uncaching.

> How is it informed?

[Documentation/filesystems/caching/netfs-api.txt]
==============
PAGE UNCACHING
==============

To uncache a page, this function should be called:

	void fscache_uncache_page(struct fscache_cookie *cookie,
				  struct page *page);

This function permits the cache to release any in-memory representation it
might be holding for this netfs page.  This function must be called once for
each page on which the read or write page functions above have been called to
make sure the cache's in-memory tracking information gets torn down.

Note that pages can't be explicitly deleted from the data file.  The whole
data file must be retired (see the relinquish cookie function below).

Furthermore, note that this does not cancel the asynchronous read or write
operation started by the read/alloc and write functions.
[/]

> Who owns the page cache in which such a page lives, the nfs client?
> Filesystem that hosts the page?  A third page cache owned by the
> cache itself?  (See my basic confusion about how many page cache
> levels you have, below.)

[Documentation/filesystems/caching/fscache.txt]
 (7) Data I/O is done direct to and from the netfs's pages.  The netfs
     indicates that page A is at index B of the data-file represented by cookie
     C, and that it should be read or written.  The cache backend may or may
     not start I/O on that page, but if it does, a netfs callback will be
     invoked to indicate completion.  The I/O may be either synchronous or
     asynchronous.
[/]

I should perhaps make the documentation more explicit: the pages passed to the
routines defined in include/linux/fscache.h are netfs pages, normally belonging
the pagecache of the appropriate netfs inode.  This is, however, mentioned in
the function banner comments in fscache.h.

> Suppose one were to take a mundane approach to the persistent cache
> problem instead of layering filesystems.  What you would do then is
> change NFS's ->write_page and variants to fiddle the persistent
> cache

It is a requirement laid down by the Linux NFS fs maintainers that the writes
to the cache be asynchronous, even if the writes to NFS aren't.

Note further that NFS's write_page() != writing to the cache.  Writing to the
cache is typically done by NFS's readpages().

Besides, at the moment, caching is suppressed for any NFS file opened for
writing due to coherency issues.  This is something to be revisited later.

> as well as the network, instead of just the network as now.

Not as now.  See above.

> This fiddling could even consist of ->write calls to another
> filesystem, though working directly with the bio interface would
> yield the fastest, and therefore to my mind, best result.

You can't necessarily access the BIO interface, and even if you can, the cache
is still a filesystem.

Essentially, what cachefiles does is to do what you say: to perform ->write
calls on another filesystem.

FS-Cache also protects the netfs against (a) there being no cache, (b) the
cache suffering a fatal I/O error and (c) the cache being removed; and protects
the cache against (d) the netfs uncaching pages that the cache is using and (e)
conflicting operations from the netfs, some of which may be queued for
asynchronous processing.

FS-Cache also groups asynchronous netfs store requests together, which
hopefully, one day, I'll be able to pass on to the backing fs.

> In any case, you find out how to write the page to backing store by
> asking the filesystem, which in the naive approach would be nfs
> augmented with caching library calls.

NFS and AFS and CIFS and ISOFS, but yes, that's what fscache is, if you like, a
caching library.

> The filesystem keeps its own metadata around to know how to map the page to
> disk.  So again naively, this metadata could tell the nfs client that the
> page is not mapped to disk at all.

The netfs should _not_ know about the metadata of a backing fs.  Firstly, there
are many different potential backing filesystems, and secondly if the netfs
knows about the metadata of the backing fs, then the backing fs has to ask the
netfs's permission if it wants to change it (background defragmentation, for
instance).

The only bit of metadata Cachefiles asks for is whether a block is represented
on disk or not.  This indicates whether the page held in that block is in the
cache or whether it has to be retrieved from the server.  The answer to that
shouldn't change if the backing fs shuffles its (meta)data around on disk.

> So I do not see what your per-page bit is for, obviously because I do not
> fully understand your caching scheme.

It's an indication to the netfs that the cache has an interest in this page,
where an interest may be a pointer to it, resources allocated or reserverd for
it, or I/O in progress upon it.

> Which I could eventually find out by reading all the patches but asking you
> is so much more fun :-)

And a waste of my time.  I've provided documentation in the main FS-Cache
patch, both as text files and in comments in header files that answer your
questions.  Please read them first.

> By the way, how many levels of page caching for the same data are
> there, is it:
> 
>   1) nfs client
>   2) cache layer's own page cache
>   3) filesystem hosting the cache
> 
> or just:
> 
>   1) nfs client page cache
>   2) filesystem hosting the cache
> 
> I think it is the second, but that is already double caching, which
> has got to hurt.

Actually, it is ideally:

    1) NFS client page cache.

But, because I can't do in-kernel O_DIRECT at the moment, with _CacheFiles_, it
is:

    1) NFS client page cache.
    2) Backing fs page cache.

With CacheFS it really is:

    1) NFS client page cache.

and it really does BIOs directly to/from the pages in the netfs.

David

Re: [PATCH 00/37] Permit filesystem local caching

[Daniel Phillips]

On Tuesday 26 February 2008 06:33, David Howells wrote:
> > Suppose one were to take a mundane approach to the persistent cache
> > problem instead of layering filesystems.  What you would do then is
> > change NFS's ->write_page and variants to fiddle the persistent
> > cache
> 
> It is a requirement laid down by the Linux NFS fs maintainers that the writes
> to the cache be asynchronous, even if the writes to NFS aren't.

As it happens, I will be hanging out for the next few days with said
NFS maintainers, it would help to be as informed as possible about
your patch set.

> Note further that NFS's write_page() != writing to the cache.  Writing to the
> cache is typically done by NFS's readpages().

Yes, of course.  But also by ->write_page no?

> > Which I could eventually find out by reading all the patches but asking you
> > is so much more fun :-)
> 
> And a waste of my time.  I've provided documentation in the main FS-Cache
> patch, both as text files and in comments in header files that answer your
> questions.  Please read them first.

37 Patches, none of which has "Documentation" in the subject line, and
you did not provide a diffstat in patch 0 for the patch set as a whole.
If I had known it was there of course I would have read it.  It is great
to see this level of documentation.  But I do not think it is fair to
blame your (one) reader for missing it.

See the smiley above?  The _real_ reason I am asking you is that I do
not think anybody understands your patch set, in spite of your
considerable efforts to address that.  Discussion in public, right or
wrong, is the only way to fix that.  It is counterproductive to drive
readers away from the discussion for fear that they may miss some point
obvious to the original author, or perhaps already discussed earlier on
lkml, and get flamed for it.

Obviously, the patch set is not going to be perfect when it goes in and
it would be a silly abuse of the open source process to require that,
but the parts where it touches the rest of the system have to be really
well understood, and it is clear from the level of participation in the
thread that they are not.

One bit that already came out of this, which you have alluded to
several times yourself but somehow seem to keep glossing over, is that
you need a ->direct_bio file operations method.  So does loopback mount.
It might be worth putting some effort into seeing how ->direct_IO can
be refactored to make that happen.  You can get it in separately on the
basis of helping loopback, and it will make your patches nicer.

Daniel

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

Daniel Phillips <phillips@phunq.net> wrote:

> > Note further that NFS's write_page() != writing to the cache.  Writing to
> > the cache is typically done by NFS's readpages().
> 
> Yes, of course.  But also by ->write_page no?

Theoretically, perhaps, but currently, no.

> 37 Patches, none of which has "Documentation" in the subject line,

Each piece of documentation is included in the patch to which it applies.
Besides, I, like everyone else, always write full documentation for interfaces
that I add, so you should have expected it to be there, right? :-)

> and you did not provide a diffstat in patch 0 for the patch set as a whole.

StGIT doesn't do that.  Besides, it's redundant information.  I'll add
something to the cover note that points out the documentation.

> One bit that already came out of this, which you have alluded to
> several times yourself but somehow seem to keep glossing over, is that
> you need a ->direct_bio file operations method.

Which I am not allowed.  I have suggested it, and it has been refused.  The
problem, as I understand it, is that this would mean BIOs external to the
filesystem which the filesystem would need additional way of keeping track of.
You have to consider that a filesystem can't rearrange any bits of itself that
have I/O in progress on them.

Furthermore, a BIO may not be appropriate.  Consider ReiserFS's tail packing,
or consider an encrypted filesystem, or, worse, a compressed filesystem.  Also,
what if the filesystem isn't backed by a blockdev?

> So does loopback mount.  It might be worth putting some effort into seeing
> how ->direct_IO can be refactored to make that happen.

Have you looked at the horrible tangle of spaghetti that is the current Linux
direct I/O model?  It would take a lot of effort to refactor it.  I've made a
couple of attempts, but the assumptions it makes make it hard.

A separate, clean, in-kernel direct-IO thing would be an easier way to go - but
it's not actually necessary at the moment.

> You can get it in separately on the basis of helping loopback, and it will
> make your patches nicer.

It will make very little difference to the code.  It would improve cachefiles's
cf-rdwr.c, yes, and it ought to improve performance.

David

Re: [PATCH 00/37] Permit filesystem local caching

[David Howells]

David Howells <dhowells@redhat.com> wrote:

> > > Have you got before/after benchmark results?
> > 
> > See attached.
> 
> Attached here are results using BTRFS (patched so that it'll work at all)
> rather than Ext3 on the client on the partition backing the cache.

And here are XFS results.

Tuning XFS makes a *really* big difference for the lots of small/medium files
being tarred case.  However, in general BTRFS is much better.

David
---


=========================
FEW BIG FILES TEST ON XFS
=========================

Completely cold caches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.286s
	user    0m0.000s
	sys     0m1.828s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.228s
	user    0m0.000s
	sys     0m1.360s

Warm NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.058s
	user    0m0.000s
	sys     0m0.060s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.122s
	user    0m0.000s
	sys     0m0.120s

Warm XFS pagecache, cold NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.181s
	user    0m0.000s
	sys     0m0.180s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m1.034s
	user    0m0.000s
	sys     0m0.404s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.540s
	user    0m0.000s
	sys     0m0.256s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.003s
	user    0m0.000s
	sys     0m0.532s


==========================================
MANY SMALL/MEDIUM FILE READING TEST ON XFS
==========================================

Completely cold caches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    4m56.827s
	user    0m0.180s
	sys     0m6.668s

Warm NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m15.084s
	user    0m0.212s
	sys     0m5.008s

Warm XFS pagecache, cold NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m13.547s
	user    0m0.220s
	sys     0m5.652s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    4m36.316s
	user    0m0.148s
	sys     0m4.440s


=======================================================
MANY SMALL/MEDIUM FILE READING TEST ON AN OPTIMISED XFS
=======================================================

mkfs.xfs -d agcount=4 -l size=128m,version=2 /dev/sda6


Completely cold caches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    3m44.033s
	user    0m0.248s
	sys     0m6.632s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    3m8.582s
	user    0m0.108s
	sys     0m3.420s

[PATCH 00/37] Permit filesystem local caching

[David Howells]

These patches add local caching for network filesystems such as NFS.

The patches can roughly be broken down into a number of sets:

  (*) 01-keys-inc-payload.diff
  (*) 02-keys-search-keyring.diff
  (*) 03-keys-callout-blob.diff

      Three patches to the keyring code made to help the CIFS people.
      Included because of patches 05-08.

  (*) 04-keys-get-label.diff

      A patch to allow the security label of a key to be retrieved.
      Included because of patches 05-08.

  (*) 05-security-current-fsugid.diff
  (*) 06-security-separate-task-bits.diff
  (*) 07-security-subjective.diff
  (*) 08-security-kernel_service-class.diff
  (*) 09-security-kernel-service.diff
  (*) 10-security-nfsd.diff

      Patches to permit the subjective security of a task to be overridden.
      All the security details in task_struct are decanted into a new struct
      that task_struct then has two pointers two: one that defines the
      objective security of that task (how other tasks may affect it) and one
      that defines the subjective security (how it may affect other objects).

      Note that I have dropped the idea of struct cred for the moment.  With
      the amount of stuff that was excluded from it, it wasn't actually any
      use to me.  However, it can be added later.

      Required for cachefiles.

  (*) 11-release-page.diff
  (*) 12-fscache-page-flags.diff
  (*) 13-add_wait_queue_tail.diff
  (*) 14-fscache.diff

      Patches to provide a local caching facility for network filesystems.

  (*) 15-cachefiles-ia64.diff
  (*) 16-cachefiles-ext3-f_mapping.diff
  (*) 17-cachefiles-write.diff
  (*) 18-cachefiles-monitor.diff
  (*) 19-cachefiles-export.diff
  (*) 20-cachefiles.diff

      Patches to provide a local cache in a directory of an already mounted
      filesystem.

  (*) 21-nfs-comment.diff
  (*) 22-nfs-fscache-option.diff
  (*) 23-nfs-fscache-kconfig.diff
  (*) 24-nfs-fscache-top-index.diff
  (*) 25-nfs-fscache-server-obj.diff
  (*) 26-nfs-fscache-super-obj.diff
  (*) 27-nfs-fscache-inode-obj.diff
  (*) 28-nfs-fscache-use-inode.diff
  (*) 29-nfs-fscache-invalidate-pages.diff
  (*) 30-nfs-fscache-iostats.diff
  (*) 31-nfs-fscache-page-management.diff
  (*) 32-nfs-fscache-read-context.diff
  (*) 33-nfs-fscache-read-fallback.diff
  (*) 34-nfs-fscache-read-from-cache.diff
  (*) 35-nfs-fscache-store-to-cache.diff
  (*) 36-nfs-fscache-mount.diff
  (*) 37-nfs-fscache-display.diff

      Patches to provide NFS with local caching.

      A couple of questions on the NFS iostat changes: (1) Should I update the
      iostat version number; (2) is it permitted to have conditional iostats?


I've massively split up the NFS patches as requested by Trond Myklebust and
Chuck Lever.  I've also brought the patches up to date with the patch window
turbulence.

--
A tarball of the patches is available at:

	http://people.redhat.com/~dhowells/fscache/patches/nfs+fscache-29.tar.bz2


To use this version of CacheFiles, the cachefilesd-0.9 is also required.  It
is available as an SRPM:

	http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9-1.fc7.src.rpm

Or as individual bits:

	http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9.tar.bz2
	http://people.redhat.com/~dhowells/fscache/cachefilesd.fc
	http://people.redhat.com/~dhowells/fscache/cachefilesd.if
	http://people.redhat.com/~dhowells/fscache/cachefilesd.te
	http://people.redhat.com/~dhowells/fscache/cachefilesd.spec

The .fc, .if and .te files are for manipulating SELinux.

David