NFS fscache offline mode?

NFS fscache offline mode?

[Tom Talpey]

Does the new NFS fscache support offline mode? That is, can
the client continue to serve cached files even in the absence
of any service communication at all?

I see that the fscache itself can do this, but it also seems to
require the netfs (NFS) to invoke it with "pinning" operations.
I don't see any pinning calls, or options to request the behavior
in the NFS code currently.

I actually have one other semi-related question. In fscache-index.c,
the fscache generates keys to match servers by computing an
{NFS version, transport protocol, port, IP address} tuple. Have
you given thought to how this might work with NFSv4.1 sessions?
With 4.1, the session allows trunking and reconnection to multiple
server addresses. It appears the cache basically won't hit on
such configurations. I think the nfs_server_key structure will
require more thought for 4.1.

Tom.

Re: NFS fscache offline mode?

[David Howells]

Tom Talpey <tmtalpey@gmail.com> wrote:

> Does the new NFS fscache support offline mode? That is, can
> the client continue to serve cached files even in the absence
> of any service communication at all?

That is not yet supported.  It's quite a tricky problem as the netfs (NFS in
this case) has to do conflict resolution when the server is contacted once
again, and that might require user interaction.

> I see that the fscache itself can do this, but it also seems to
> require the netfs (NFS) to invoke it with "pinning" operations.
> I don't see any pinning calls, or options to request the behavior
> in the NFS code currently.

Pinning and reservations are not yet implemented in the cache.  I've been
holding off on implementing them on the basis that until the code gets
upstream, I have to expect that I might have to make substantial alterations
to the code I do have.

In any case, offline mode isn't something that FS-Cache itself cares about.
It is purely a data store.  The network filesystems using it must implement
offline mode.

> I actually have one other semi-related question. In fscache-index.c, the
> fscache generates keys to match servers by computing an {NFS version,
> transport protocol, port, IP address} tuple.  Have you given thought to how
> this might work with NFSv4.1 sessions?  With 4.1, the session allows
> trunking and reconnection to multiple server addresses. It appears the cache
> basically won't hit on such configurations. I think the nfs_server_key
> structure will require more thought for 4.1.

I've been thinking for a while about how to map multiple server addresses onto
one cache for servers that all serve the same data.  It's not clear how to do
it, since, as far as I know, there's no way to automatically work out that two
servers should be treated as being the same.

With AFS it's easier, since the volumes are defined by the cell they're in,
not by the servers that are serving them.  As far as I know NFS doesn't do
things that way.

One thing I have wondered about is sticking aliases in the cache (symlinks or
whatever) when an NFS mount is made that has a list of server addresses.
However, this assumes:

 (1) The file handles on one server match those of another server serving the
     same set of files.

 (2) If two servers are serving the same data, then overlapping exports match
     completely.

I feel there's another problem with aliases like this in the case of the
address a server that's been added as an alias being reused to server
different data.  What I think I need is consistency data from the server about
the server, so that I can tell that the server is no longer what I thought it
was.


In fact, it might be possible to hide the fact that there are aliases from
FS-Cache entirely.  Say that the NFS client is given a list of IP addresses
that are all serving the same data.  It then uses the lowest IP address it is
given as the key to FS-Cache, no matter which server it is actually talking
to.

David

RE: NFS fscache offline mode?

[Muntz, Daniel]

 

> -----Original Message-----
> From: David Howells [mailto:dhowells@redhat.com] 
> Sent: Thursday, April 09, 2009 4:15 AM
> To: Tom Talpey
> Cc: dhowells@redhat.com; linux-nfs@vger.kernel.org
> Subject: Re: NFS fscache offline mode?
> 
> Tom Talpey <tmtalpey@gmail.com> wrote:
> 
> > Does the new NFS fscache support offline mode? That is, can 
> the client 
> > continue to serve cached files even in the absence of any service 
> > communication at all?
> 
> That is not yet supported.  It's quite a tricky problem as 
> the netfs (NFS in this case) has to do conflict resolution 
> when the server is contacted once again, and that might 
> require user interaction.

It's not _so_ tricky for the read-only case, and is quite useful (see
papers by Huston+Honeyman re: disconnected AFS).  For writes, my
recollection is that you pretty much need human intervention to resolve
conflicts unless a trivial resolution mechanism is defined.

> 
> > I see that the fscache itself can do this, but it also seems to 
> > require the netfs (NFS) to invoke it with "pinning" operations.
> > I don't see any pinning calls, or options to request the 
> behavior in 
> > the NFS code currently.
> 
> Pinning and reservations are not yet implemented in the 
> cache.  I've been holding off on implementing them on the 
> basis that until the code gets upstream, I have to expect 
> that I might have to make substantial alterations to the code 
> I do have.
> 
> In any case, offline mode isn't something that FS-Cache 
> itself cares about.
> It is purely a data store.  The network filesystems using it 
> must implement offline mode.

That's a bit flippant.  You could just as well say that the "network"
file system should implement its own disk cache.  If you're going to
cache under the fs, you could also do a lot to hide the disconnected
status from the fs, once again, not too bad if you're doing read-only.

BTW: could we not use the term "network" file system when discussing
FS-Cache?  You've also mentioned using it for iso9660, so it's not just
for network file systems.

> 
> > I actually have one other semi-related question. In 
> fscache-index.c, 
> > the fscache generates keys to match servers by computing an {NFS 
> > version, transport protocol, port, IP address} tuple.  Have 
> you given 
> > thought to how this might work with NFSv4.1 sessions?  With 
> 4.1, the 
> > session allows trunking and reconnection to multiple server 
> addresses. 
> > It appears the cache basically won't hit on such configurations. I 
> > think the nfs_server_key structure will require more 
> thought for 4.1.
> 
> I've been thinking for a while about how to map multiple 
> server addresses onto one cache for servers that all serve 
> the same data.  It's not clear how to do it, since, as far as 
> I know, there's no way to automatically work out that two 
> servers should be treated as being the same.
> 
> With AFS it's easier, since the volumes are defined by the 
> cell they're in, not by the servers that are serving them.  
> As far as I know NFS doesn't do things that way.
> 
> One thing I have wondered about is sticking aliases in the 
> cache (symlinks or
> whatever) when an NFS mount is made that has a list of server 
> addresses.
> However, this assumes:
> 
>  (1) The file handles on one server match those of another 
> server serving the
>      same set of files.
> 
>  (2) If two servers are serving the same data, then 
> overlapping exports match
>      completely.
> 
> I feel there's another problem with aliases like this in the 
> case of the address a server that's been added as an alias 
> being reused to server different data.  What I think I need 
> is consistency data from the server about the server, so that 
> I can tell that the server is no longer what I thought it was.
> 
> 
> In fact, it might be possible to hide the fact that there are 
> aliases from FS-Cache entirely.  Say that the NFS client is 
> given a list of IP addresses that are all serving the same 
> data.  It then uses the lowest IP address it is given as the 
> key to FS-Cache, no matter which server it is actually talking to.
> 
> David
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-nfs" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> 

Re: NFS fscache offline mode?

[David Howells]

Muntz, Daniel <Dan.Muntz@netapp.com> wrote:

> It's not _so_ tricky for the read-only case, and is quite useful (see
> papers by Huston+Honeyman re: disconnected AFS).  For writes, my
> recollection is that you pretty much need human intervention to resolve
> conflicts unless a trivial resolution mechanism is defined.

Oh, I agree that's it's much simpler for read-only, and that there's no
trickiness for files that haven't been modified locally whilst offline; but if
any have, it could then be a problem.

The simplest way is to simply discard local changes upon reconnection if the
backing file has changed.  Isn't that what we do anyway when in connected
mode.

> > In any case, offline mode isn't something that FS-Cache itself cares
> > about.  It is purely a data store.  The network filesystems using it must
> > implement offline mode.
> 
> That's a bit flippant.  You could just as well say that the "network" file
> system should implement its own disk cache.  If you're going to cache under
> the fs, you could also do a lot to hide the disconnected status from the fs,
> once again, not too bad if you're doing read-only.

I disagree.  FS-Cache provides or will provide the local storage required to
perform offline operation, the ability to pin data within that storage, and
the ability to tag the stored data with the metadata required; but in my
opinion, the netfs, be it NFS, AFS or CIFS, must do the actual work of:

 (1) configuring what must be available for disconnected operation,

 (2) pulling files into the cache in preparation,

 (3) handling requests from userspace that can't be satisfied when offline,
     and

 (4) resolving conflicts afterwards.

The netfs knows about the structure of the filesystem; fscache does not.  The
user interacts with the netfs, not directly with fscache.

> BTW: could we not use the term "network" file system when discussing
> FS-Cache?  You've also mentioned using it for iso9660, so it's not just
> for network file systems.

All my documents, code and emails are laced with the term 'netfs' to mean a
client of FS-Cache.

As I have pointed out in my documentation, whilst iso9660 isn't a network
filesystem, it can be considered in the same group for this topic.

David

RE: NFS fscache offline mode?

[Muntz, Daniel]

 

> -----Original Message-----
> From: David Howells [mailto:dhowells@redhat.com] 
> Sent: Thursday, April 09, 2009 2:22 PM
> To: Muntz, Daniel
> Cc: dhowells@redhat.com; Tom Talpey; linux-nfs@vger.kernel.org
> Subject: Re: NFS fscache offline mode?
> 
> Muntz, Daniel <Dan.Muntz@netapp.com> wrote:
> 
> > It's not _so_ tricky for the read-only case, and is quite 
> useful (see 
> > papers by Huston+Honeyman re: disconnected AFS).  For writes, my 
> > recollection is that you pretty much need human intervention to 
> > resolve conflicts unless a trivial resolution mechanism is defined.
> 
> Oh, I agree that's it's much simpler for read-only, and that 
> there's no trickiness for files that haven't been modified 
> locally whilst offline; but if any have, it could then be a problem.
> 
> The simplest way is to simply discard local changes upon 
> reconnection if the backing file has changed.  Isn't that 
> what we do anyway when in connected mode.

The two trivial cases are: discard changes from disconnected mode, or
propagate all changes when reconnected.  Doing something more
intelligent is difficult.  If writes are allowed, you still run the risk
of evicting something you really wanted in the cache (oops, forgot to
pin my .bashrc), or getting unexpected disk-full errors.  But, you still
have something potentially useful if you just make the cache ro when
disconnected.

> > > In any case, offline mode isn't something that FS-Cache 
> itself cares 
> > > about.  It is purely a data store.  The network 
> filesystems using it 
> > > must implement offline mode.
> > 
> > That's a bit flippant.  You could just as well say that the 
> "network" 
> > file system should implement its own disk cache.  If you're 
> going to 
> > cache under the fs, you could also do a lot to hide the 
> disconnected 
> > status from the fs, once again, not too bad if you're doing 
> read-only.
> 
> I disagree.  FS-Cache provides or will provide the local 
> storage required to perform offline operation, the ability to 
> pin data within that storage, and the ability to tag the 
> stored data with the metadata required; but in my opinion, 
> the netfs, be it NFS, AFS or CIFS, must do the actual work of:
> 
>  (1) configuring what must be available for disconnected operation,

You've captured the data and metadata.  I suppose there might be some
fs-specific hooks needed (e.g., something needed to prevent the nfs
client from thinking the server is down).  IMHO, it would be really
handy if a canonical list of such hooks could be made, and FS-Cache
could handle these similar to how export ops work in NFS--the fs
supplies routines that FS-Cache calls to make disconnected operation
work.

> 
>  (2) pulling files into the cache in preparation,

Some trivial fs-agnostic ways to do this: use the system before
disconnecting, or run a process to walk through the fs catting files
that you want in disconnected mode (pulling them into the cache).  Just
out of curiosity, could one 'tar' up the cache on one machine and drop
it on another and have it work?  There must be enough state on disk so
that a reboot doesn't invalidate your cache, so I'm guessing this might
be another way to warm your cache.

>
 
>  (3) handling requests from userspace that can't be satisfied 
> when offline,

What did you have in mind?

>      and
> 
>  (4) resolving conflicts afterwards.

If you stick to the ro case, or the two simple conflict resolution
schemes, then this is not an issue.

> 
> The netfs knows about the structure of the filesystem; 
> fscache does not.  The user interacts with the netfs, not 
> directly with fscache.
> 
> > BTW: could we not use the term "network" file system when 
> discussing 
> > FS-Cache?  You've also mentioned using it for iso9660, so it's not 
> > just for network file systems.
> 
> All my documents, code and emails are laced with the term 
> 'netfs' to mean a client of FS-Cache.

That's what 'sed' is for :-)

> 
> As I have pointed out in my documentation, whilst iso9660 
> isn't a network filesystem, it can be considered in the same 
> group for this topic.
> 
> David
> 

Re: NFS fscache offline mode?

[David Howells]

Muntz, Daniel <Dan.Muntz@netapp.com> wrote:

> >  (1) configuring what must be available for disconnected operation,
> 
> You've captured the data and metadata.  I suppose there might be some
> fs-specific hooks needed (e.g., something needed to prevent the nfs
> client from thinking the server is down).  IMHO, it would be really
> handy if a canonical list of such hooks could be made, and FS-Cache
> could handle these similar to how export ops work in NFS--the fs
> supplies routines that FS-Cache calls to make disconnected operation
> work.

No, disconnected operation is a netfs thing, not an FS-Cache thing, I think.
Take NFS for example: it must know that it is in disconnected mode; it can't
fling GETATTR requests at the server to determine whether a file is valid; it
can't ask for a lease on that file; it must rely on FS-Cache to have the data
and metadata it requires to make files and directories available.

Similarly, for AFS, it can't go to the server to validate the vnodes it has
stored, and it can't request callbacks on the files it opens.  In fact both
filesystems probably should return their leases/callbacks when entering
disconnected mode.

FS-Cache doesn't do anything but store metadata and data NFS gets from the
server as it arrives, and retrieve or invalidate that data and metadata when
asked by NFS.

For the most part, NFS and AFS operate the following path in any request:

	if (in_fscache(data_or_metadata)) {
		retrieve_from_fscache(data_or_metadata);
	} else {
		get_from_server(data_or_metadata);
		store_to_fscache(data_or_metadata);
	}

But with the advent of disconnected mode, that becomes:

	if (in_fscache(data_or_metadata)) {
		retrieve_from_fscache(data_or_metadata);
	} else if (in_disconnected_mode) {
		return -EDISCONNECTED;
	} else {
		get_from_server(data_or_metadata);
		store_to_fscache(data_or_metadata);
	}

If FS-Cache doesn't have a file, directory, page or dirent stored, then the
netfs returns EDISCONNECTED or whatever for that item.

> >  (2) pulling files into the cache in preparation,
> 
> Some trivial fs-agnostic ways to do this: use the system before
> disconnecting, or run a process to walk through the fs catting files
> that you want in disconnected mode (pulling them into the cache).

Yeah, a userspace solution would be simplest here:

	[/usr/sbin/use-in-disconnected-mode]
	#!/bin/sh
	tar cf - $* >/dev/zero

However, we'd want to pin all those files, so using tar isn't necessarily the
best.  Pinning could be achieved by opening each file, emitting an fadvise()
or ioctl() to say that this should be pinned, and then reading the first byte
of each page in the file.

> Just out of curiosity, could one 'tar' up the cache on one machine and drop
> it on another and have it work?  There must be enough state on disk so that
> a reboot doesn't invalidate your cache, so I'm guessing this might be
> another way to warm your cache.

Perhaps.  You'd have to supply --xattr to tar, but you could do it.

> >  (3) handling requests from userspace that can't be satisfied 
> > when offline,
> 
> What did you have in mind?

Someone tries to access a file that isn't in the cache, say.  FS-Cache says
'no I haven't got that' and then the netfs needs to do something with it -
even if that's just returning FS-Cache's error (-ENODATA) to the caller.

> >      and
> > 
> >  (4) resolving conflicts afterwards.
> 
> If you stick to the ro case, or the two simple conflict resolution
> schemes, then this is not an issue.

Mostly true.  But it's still dealt with by the netfs.

For the discard-on-conflict option, with NFS and AFS it'd be a case of compare
an inode that's in core (if the inode is in use) or in the cache next time it
is accessed with what's on the server - as is done in connected operation.

For the writeback-on-conflict option, it's not a non-issue.  The netfs would
have to write back what's in the pagecache or in FS-Cache, and that's
something we don't do now.

David