asynchronous locks for cluster exports

asynchronous locks for cluster exports

[J. Bruce Fields]

We'd like an asynchronous posix locking interface so that we can provide NFS
clients with cluster-coherent locking without blocking lockd while the
filesystem goes off and talks to other nodes.

So, here's one attempt.  It also includes an draft implementation of the
filesystem side for GFS2, which may well be wrong--it's not even tested
yet--but hopefully gives some idea what would be necessary.

A few points, among others, that I'm unsure of:

	- We added a new ->lock() export operation, figuring this was a feature
	  that only lockd and nfsd care about for now, and that we'd rather not
	  muck about with common locking code.  But the export operation is
	  pretty much identical to the file ->lock() operation; would it make
	  more sense to use that?

	- The filesystem returns the lock results to lockd using the
	  ->fl_notify() callback.  We add a few arguments to fl_notify() to
	  pass the results, and add a return value so the filesystem can
	  recognize the case where the callback comes after lockd has given up
	  waiting and returned an error to the user.  Presumably the filesystem
	  needs to have a way to cancel the lock in this case.  (Our GFS code
	  ignores this problem for now.)  Maybe it would be better to just poke
	  lockd when the result is ready and let it discover what happened by
	  retrying the original ->lock() call?  Or maybe we should use a
	  separate callback?

	- We're ignoring the blocking lock case for now under the assumption
	  it's always OK for lockd to return an immediate "denied" in that
	  case, then use the granted callback, even in cases where it doesn't
	  know for sure that there's a conflicting lock.

Thoughts?  Better ideas?

--b.

Re: asynchronous locks for cluster exports

[Christoph Hellwig]

On Wed, Dec 06, 2006 at 12:34:10AM -0500, J. Bruce Fields wrote:
> We'd like an asynchronous posix locking interface so that we can provide NFS
> 	- We added a new ->lock() export operation, figuring this was a feature
> 	  that only lockd and nfsd care about for now, and that we'd rather not
> 	  muck about with common locking code.  But the export operation is
> 	  pretty much identical to the file ->lock() operation; would it make
> 	  more sense to use that?

This definitly needs to be merged back into the ->lock file operation

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: asynchronous locks for cluster exports

[J. Bruce Fields]

On Thu, Dec 07, 2006 at 04:51:08PM +0000, Christoph Hellwig wrote:
> On Wed, Dec 06, 2006 at 12:34:10AM -0500, J. Bruce Fields wrote:
> > We'd like an asynchronous posix locking interface so that we can provide NFS
> > 	- We added a new ->lock() export operation, figuring this was a feature
> > 	  that only lockd and nfsd care about for now, and that we'd rather not
> > 	  muck about with common locking code.  But the export operation is
> > 	  pretty much identical to the file ->lock() operation; would it make
> > 	  more sense to use that?
> 
> This definitly needs to be merged back into the ->lock file operation

So the interesting question is whether we can merge the semantics in a
reasonable way.  The export operation implemented by the current version
of these patches returns more or less immediately with success, -EAGAIN,
or -EINPROGRESS; in the latter case the filesystem later calls fl_notify
to communicate the results.  The existing file lock operation normally
blocks until the lock is actually granted.

The one file lock operation could do both, and just switch between the
two cases depending on whether fl_notify is defined.  Would the
semantics be clear enough?

I find the existing use of ->lock() a little odd as it is; stuff like this,
from fcntl_getlk():

	if (filp->f_op && filp->f_op->lock) {
		error = filp->f_op->lock(filp, F_GETLK, &file_lock);
		if (file_lock.fl_ops && file_lock.fl_ops->fl_release_private)
			file_lock.fl_ops->fl_release_private(&file_lock);
		if (error < 0)
			goto out;
		else
			fl = (file_lock.fl_type == F_UNLCK ? NULL : &file_lock);
	} else {
		fl = (posix_test_lock(filp, &file_lock, &cfl) ? &cfl : NULL);
	}

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: asynchronous locks for cluster exports

[J. Bruce Fields]

On Wed, Dec 06, 2006 at 12:34:10AM -0500, J. Bruce Fields wrote:
> We'd like an asynchronous posix locking interface so that we can provide NFS
> clients with cluster-coherent locking without blocking lockd while the
> filesystem goes off and talks to other nodes.
> 
> So, here's one attempt.  It also includes an draft implementation of the
> filesystem side for GFS2, which may well be wrong--it's not even tested
> yet--but hopefully gives some idea what would be necessary.

By the way, I'm keeping updated versions of this patch series in the
server-cluster-locking-api branch of the repository at
git://linux-nfs.org/~bfields/linux.git.

So if you already have a git tree sitting around (cloned with something
like "git clone
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git")

They you should be able to get this with

	git fetch git://linux-nfs.org/~bfields/linux.git +server-cluster-locking-api:server-cluster-locking-api
	git checkout server-cluster-locking-api

Alternatively you can browse the patches without git from

	http://linux-nfs.org/cgi-bin/gitweb.cgi?p=bfields-2.6.git;a=shortlog;h=server-cluster-locking-api

--b.

asynchronous locks for cluster exports

[J. Bruce Fields]

This is another attempt at a posix locking interface that allows us to
provide NFS clients with cluster-coherent locking without blocking lockd
while the filesystem goes off and talks to other nodes.

This time we're using the existing file ->lock operation instead of
defining a new export operation for the purpose, and factoring out the
code that switches between calling ->lock or the default local
operation, so that lock managers and local callers go through the same
code.

I find the existing interfaces a little odd, and while I think I managed
to combine them correctly, I'd certainly appreciate any review.

The gfs2 implementation in the last patch is (unfortunately) still just
a rough draft that needs some more thought and some testing.

--b.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

asynchronous locks for cluster exports

[J. Bruce Fields]

This is another attempt at a posix locking interface that allows us to
provide NFS clients with cluster-coherent locking without blocking lockd
while the filesystem goes off and talks to other nodes.

This time we're using the existing file ->lock operation instead of
defining a new export operation for the purpose, and factoring out the
code that switches between calling ->lock or the default local
operation, so that lock managers and local callers go through the same
code.

I find the existing interfaces a little odd, and while I think I managed
to combine them correctly, I'd certainly appreciate any review.

The gfs2 implementation in the last patch is (unfortunately) still just
a rough draft that needs some more thought and some testing.

--b.

Re: asynchronous locks for cluster exports

[Christoph Hellwig]

On Sat, Feb 03, 2007 at 12:30:55AM -0500, J. Bruce Fields wrote:
> The gfs2 implementation in the last patch is (unfortunately) still just
> a rough draft that needs some more thought and some testing.

So what exactly in this patch is tested?

Re: asynchronous locks for cluster exports

[J. Bruce Fields]

On Sat, Feb 03, 2007 at 08:39:29AM +0000, Christoph Hellwig wrote:
> On Sat, Feb 03, 2007 at 12:30:55AM -0500, J. Bruce Fields wrote:
> > The gfs2 implementation in the last patch is (unfortunately) still just
> > a rough draft that needs some more thought and some testing.
> 
> So what exactly in this patch is tested?

Marc's testing this stuff against gpfs.  Yeah, I know, I know.  I'm not
asking this be merged until it makes sense purely from the point of view
of in-tree users.

Right now all we know is that it doesn't appear to break local or NFS
locking, that it works for one out-of-tree filesystem, and that it looks
like it should be right for gfs--but the gfs implementation is just a
sketch, totally untested.  I'm here at connectathon working with one of
the gfs developers, so hopefully that should change soon.

We'd like to work with ocfs too at some point, but last I heard they
hadn't yet tried to implement cluster-coherent posix locking.

--b.

Re: asynchronous locks for cluster exports

[J. Bruce Fields]

On Sat, Feb 03, 2007 at 12:30:55AM -0500, J. Bruce Fields wrote:
> 
> This is another attempt at a posix locking interface that allows us to
> provide NFS clients with cluster-coherent locking without blocking lockd
> while the filesystem goes off and talks to other nodes.

Marc and I have an updated version of this at:

	git://linux-nfs.org/~bfields/linux.git

(See the server-cluster-locking-api branch.)

Changes include:

	- bugfixes for GFS2; thanks to some setup help from Wendy Chang
	  and others, Marc has been able to run locking tests against an
	  nfs-exported GFS2 filesystem
	- preallocation of storage for conflicting lock in lockd's
	  testlock implementation, to avoid a race identified by Trond
	- Cleanup suggested by Christoph and others, including:
		- creation of posix-to-flock helper functions
		- rewrite of posix_test_lock interface to agree with
		  ->lock( ,F_GETLK, )
		- removal of some unnecessary parentheses, untangling of
		  some slightly tortured logic

We're hoping to get a detailed review from Trond sometime in the coming
month, after which we'll probably mailbomb linux-fsdevel again, but
any comments are welcome in the meantime.

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs