PATCH [0/10] lease interfaces for version 4 NFSD

PATCH [0/10] lease interfaces for version 4 NFSD

[William A.(Andy) Adamson]

The following patches provide an interface to the lease subsystem in the 
current VFS locking code.

The version 4 nfsd delegation implementation uses leases to co-ordinate 
behavior between local, samba, and nfs access.

The main design points are

	- Seperate the file_lock processing from the file descriptor processing in 
fcntl_setlease.
	- New non-file descriptor interface to setlease
	- Use new lock manager callbacks to allow nfsd to create, break, and release 
leases.
	- Provide default lock manager callbacks for fcntl_setlease

-->Andy Adamson

file leases

[Jamie Lokier]

William A.(Andy) Adamson wrote:
> The version 4 nfsd delegation implementation uses leases to co-ordinate 
> behavior between local, samba, and nfs access.

Are they working properly?

When I share a directory using Samba 3.0.6, sometimes local access to
a file stalls for a while - on the order of 10 seconds to open a file.

It is very annoying, and I had to turn off oplocks in order to
reasonably edit files in a shared directory.  I presume Samba
implements oplocks using leases, and that is the cause of local access
stalling.

I wonder if leases aren't working as they should be, or if Samba isn't
using them properly.

-- Jamie

Re: file leases

[William A.(Andy) Adamson]

> William A.(Andy) Adamson wrote:
> > The version 4 nfsd delegation implementation uses leases to co-ordinate 
> > behavior between local, samba, and nfs access.
> 
> Are they working properly?

in the testing that i have done, yes, they work.

> 
> When I share a directory using Samba 3.0.6, sometimes local access to
> a file stalls for a while - on the order of 10 seconds to open a file.

if your open() is blocking (did not specify O_NONBLOCK) then break_lease will 
contact all other processes (samba clients) one at a time, and will not return 
until all have been contacted. this could take a while.

> 
> It is very annoying, and I had to turn off oplocks in order to
> reasonably edit files in a shared directory.  I presume Samba
> implements oplocks using leases, and that is the cause of local access
> stalling.
> 
> I wonder if leases aren't working as they should be, or if Samba isn't
> using them properly.

i haven't run the samba server. 

> 
> -- Jamie
> 

Re: file leases

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On Mon, 20 Sep 2004 16:34:30 -0400 "William A.(Andy) Adamson" <andros@citi.umich.edu> wrote:
>
> > When I share a directory using Samba 3.0.6, sometimes local access to
> > a file stalls for a while - on the order of 10 seconds to open a file.
> 
> if your open() is blocking (did not specify O_NONBLOCK) then break_lease will 
> contact all other processes (samba clients) one at a time, and will not return 
> until all have been contacted. this could take a while.

And, of course, the samba servers may have to contact their clients and
get them to release their oplocks ...

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: file leases

[Jamie Lokier]

Stephen Rothwell wrote:
> > > When I share a directory using Samba 3.0.6, sometimes local access to
> > > a file stalls for a while - on the order of 10 seconds to open a file.
> > 
> > if your open() is blocking (did not specify O_NONBLOCK) then
> > break_lease will contact all other processes (samba clients) one
> > at a time, and will not return until all have been contacted. this
> > could take a while.
> 
> And, of course, the samba servers may have to contact their clients and
> get them to release their oplocks ...

Quite, but the client is a Windows 2000 box completely idle with
nothing holding the file open, and a good LAN connection.

If an idle client won't release oplocks quickly on request, then
there's a problem.  Surely people using lots of Windows clients don't
have to wait 10 seconds or more to open a file which somebody else had
open recently on a different box?  I suspect a Samba problem but
anyways the answer is to disable oplocks.  It _could_ be a fault in
leases, which is why I asked if they're known to be working.

-- Jamie

Re: file leases

[William A.(Andy) Adamson]

> Stephen Rothwell wrote:
> > > > When I share a directory using Samba 3.0.6, sometimes local access to
> > > > a file stalls for a while - on the order of 10 seconds to open a file.
> > > 
> > > if your open() is blocking (did not specify O_NONBLOCK) then
> > > break_lease will contact all other processes (samba clients) one
> > > at a time, and will not return until all have been contacted. this
> > > could take a while.
> > 
> > And, of course, the samba servers may have to contact their clients and
> > get them to release their oplocks ...
> 
> Quite, but the client is a Windows 2000 box completely idle with
> nothing holding the file open, and a good LAN connection.
> 
> If an idle client won't release oplocks quickly on request, then
> there's a problem.  

it's not a bug, it's a feature! the open from the idle windows box is stalled 
at the server while the server sends break_lease messages to all the other 
clients with op locks on the file. that is the promise of op locks.

> Surely people using lots of Windows clients don't
> have to wait 10 seconds or more to open a file which somebody else had
> open recently on a different box?  I suspect a Samba problem but
> anyways the answer is to disable oplocks.  It _could_ be a fault in
> leases, which is why I asked if they're known to be working.
> 
> -- Jamie
> 

Re: file leases

[Jamie Lokier]

William A.(Andy) Adamson wrote:
> > Quite, but the client is a Windows 2000 box completely idle with
> > nothing holding the file open, and a good LAN connection.
> > 
> > If an idle client won't release oplocks quickly on request, then
> > there's a problem.  
> 
> it's not a bug, it's a feature! the open from the idle windows box
> is stalled at the server while the server sends break_lease messages
> to all the other clients with op locks on the file. that is the
> promise of op locks.

Eh?  There's only one client, and it's idle.  Presumably it's holding
an oplock.

When I want to edit a file directly on the server, opening for write
stalls for a long time.  Presumably the lease causes Samba to send a
break_lease to the _one_ client.  Being an idle client that's not
holding the file open, it should respond immediately to break_lease,
and then Samba should let me open the file locally.

That's how leases are supposed to work, but something in that whole
arrangement isn't working properly.  Samba is 3.0.6-r3, kernel is
2.6.8 (both gentoo versions).

-- Jamie

Re: file leases

[Jeremy Allison]

On Tue, Sep 21, 2004 at 01:44:59PM +0100, Jamie Lokier wrote:
> William A.(Andy) Adamson wrote:
> > > Quite, but the client is a Windows 2000 box completely idle with
> > > nothing holding the file open, and a good LAN connection.
> > > 
> > > If an idle client won't release oplocks quickly on request, then
> > > there's a problem.  
> > 
> > it's not a bug, it's a feature! the open from the idle windows box
> > is stalled at the server while the server sends break_lease messages
> > to all the other clients with op locks on the file. that is the
> > promise of op locks.
> 
> Eh?  There's only one client, and it's idle.  Presumably it's holding
> an oplock.
> 
> When I want to edit a file directly on the server, opening for write
> stalls for a long time.  Presumably the lease causes Samba to send a
> break_lease to the _one_ client.  Being an idle client that's not
> holding the file open, it should respond immediately to break_lease,
> and then Samba should let me open the file locally.
> 
> That's how leases are supposed to work, but something in that whole
> arrangement isn't working properly.  Samba is 3.0.6-r3, kernel is
> 2.6.8 (both gentoo versions).

This is easy to check. Turn on debug level 10 with millisecond log
timestamps and watch the whole process from the smbd point of view.
What is stalling should become obvious.

Jeremy.

Re: file leases

[Jamie Lokier]

Jeremy Allison wrote:
> > When I want to edit a file directly on the server, opening for write
> > stalls for a long time.  Presumably the lease causes Samba to send a
> > break_lease to the _one_ client.  Being an idle client that's not
> > holding the file open, it should respond immediately to break_lease,
> > and then Samba should let me open the file locally.
> > 
> > That's how leases are supposed to work, but something in that whole
> > arrangement isn't working properly.  Samba is 3.0.6-r3, kernel is
> > 2.6.8 (both gentoo versions).
> 
> This is easy to check. Turn on debug level 10 with millisecond log
> timestamps and watch the whole process from the smbd point of view.
> What is stalling should become obvious.

Ok, I've just observed a 45 second stall with log level = 10.  I
forgot to set millisecond timestamps but I don't think that's a a
problem.

Here's the command which stalls.  It's on a local filesystem on the
server, Linux 2.6.8, Samba 3.0.6:

jamie@kushida ~/x $ date; cp -p 3rd.jpg 4th.jpg; date
Wed Sep 22 11:50:49 BST 2004
Wed Sep 22 11:51:34 BST 2004

This was triggered by creating 3rd.jpg (copied locally from
elsewhere).  Then viewing it in Internet Explorer on the Windows ME
client.  Then closing IE, and trying to copy the file locally.

Looking at the log, I see Samba gets the lease and grants the oplock
to the Window client:

[2004/09/22 11:50:40, 3] smbd/oplock_linux.c:linux_set_kernel_oplock(185)
  linux_set_kernel_oplock: got kernel oplock on file x/3rd.jpg, dev = 303, inode = 1325168, file_id = 7
[2004/09/22 11:50:40, 5] smbd/oplock.c:set_file_oplock(218)
  set_file_oplock: granted oplock on file x/3rd.jpg, dev = 303, inode = 1325168, file_id = 7, tv_sec = 41515900, tv_usec = 68af0

At about 11:50:49 I expect to see linux_oplock_receive_message and
then linux_release_kernel_oplock -- because I see these at times when
there's no stall -- but I *don't* see either in the Samba log after
this point:

kushida ~ # grep linux_ /var/log/samba/log.smbd
......
[2004/09/22 11:38:59, 3] smbd/oplock_linux.c:linux_set_kernel_oplock(185)
  linux_set_kernel_oplock: got kernel oplock on file x/3rd.jpg, dev = 303, inode = 1325168, file_id = 5
[2004/09/22 11:39:08, 10] smbd/oplock_linux.c:linux_release_kernel_oplock(196)
  linux_release_kernel_oplock: file x/3rd.jpg, dev = 303, inode = 1325168 file_id = 5 has kernel oplock state of 1.
[2004/09/22 11:39:08, 3] smbd/oplock_linux.c:linux_set_kernel_oplock(185)
  linux_set_kernel_oplock: got kernel oplock on file x/3rd.jpg, dev = 303, inode = 1325168, file_id = 6
[2004/09/22 11:39:10, 3] smbd/oplock_linux.c:linux_oplock_receive_message(149)
  linux_oplock_receive_message: kernel oplock break request received for dev = 303, inode = 1325168 fd = 28, fileid = 6
[2004/09/22 11:39:10, 3] smbd/oplock_linux.c:linux_kernel_oplock_parse(239)
[2004/09/22 11:39:10, 10] smbd/oplock_linux.c:linux_release_kernel_oplock(196)
  linux_release_kernel_oplock: file x/3rd.jpg, dev = 303, inode = 1325168 file_id = 6 has kernel oplock state of 0.
[2004/09/22 11:50:40, 3] smbd/oplock_linux.c:linux_set_kernel_oplock(185)
  linux_set_kernel_oplock: got kernel oplock on file x/3rd.jpg, dev = 303, inode = 1325168, file_id = 7
kushida ~ #

Perhaps more worryingly, there is no communication with Windows ME, so
Windows ME may still think it has the oplock.  20 minutes later, after
viewing the file a couple more times in ME, and after editing it
locally, there are no further messages with "oplock" or "linux" in the
Samba log.

30 minutes later I edit the file in Windows ME with notepad, writing
to it.  That breaks the lease because Samba breaks it itself; then it
reacquires the lease after writing the file.


It looks like either:

  (1) Linux leases aren't working: they're not signalling Samba.

  (2) or, Samba is not interfacing with Linux leases properly, so
      is missing the signal.


-- Jamie

Re: file leases

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 1902 bytes --]

On Wed, 22 Sep 2004 12:32:32 +0100 Jamie Lokier <jamie@shareable.org> wrote:
>
> Ok, I've just observed a 45 second stall with log level = 10.  I
> forgot to set millisecond timestamps but I don't think that's a a
> problem.

This 45 seconds is, of course, the default lease break time in the kernel
i.e. this is the amount of time the kernel gives the lease owner to
relinquish the lease after which it unilaterally just breaks the lease.

> This was triggered by creating 3rd.jpg (copied locally from
> elsewhere).  Then viewing it in Internet Explorer on the Windows ME
> client.  Then closing IE, and trying to copy the file locally.

Closing IE, won't cause the Windows client to release the lease, it tries
to hold it as long as possible so that it knows its local cached copy of
the file is still correct.

> At about 11:50:49 I expect to see linux_oplock_receive_message and
> then linux_release_kernel_oplock -- because I see these at times when
> there's no stall -- but I *don't* see either in the Samba log after
> this point:

So this is not consistent i.e. it works as expected most of the time?

> Perhaps more worryingly, there is no communication with Windows ME, so
> Windows ME may still think it has the oplock.  20 minutes later, after
> viewing the file a couple more times in ME, and after editing it
> locally, there are no further messages with "oplock" or "linux" in the
> Samba log.

This makes sense as Windows still thinks it has the oplock.

> It looks like either:
> 
>   (1) Linux leases aren't working: they're not signalling Samba.
> 
>   (2) or, Samba is not interfacing with Linux leases properly, so
>       is missing the signal.

Either is possible.  Is this a standard kernel.org 2.6.8 kernel and
a standard samba 3.0.6 - or are these distribution versions?

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: file leases

[Jamie Lokier]

Stephen Rothwell wrote:
> > This was triggered by creating 3rd.jpg (copied locally from
> > elsewhere).  Then viewing it in Internet Explorer on the Windows ME
> > client.  Then closing IE, and trying to copy the file locally.
> 
> Closing IE, won't cause the Windows client to release the lease, it tries
> to hold it as long as possible so that it knows its local cached copy of
> the file is still correct.

Yes, that's what I assumed.

> > At about 11:50:49 I expect to see linux_oplock_receive_message and
> > then linux_release_kernel_oplock -- because I see these at times when
> > there's no stall -- but I *don't* see either in the Samba log after
> > this point:
> 
> So this is not consistent i.e. it works as expected most of the time?

It's not consistent.  Most of the time it stalls with oplocks enabled
in Samba.

Specifically: If I have been viewing or editing a file in Windows,
then most of the time oplocks viewing or editing the same file on the
server stalls.  I thought it was stalling every time.

However when I did these tests today with the "cp" command, I was
surprised to see it not stalling sometimes.  So it is inconsistent.

> > Perhaps more worryingly, there is no communication with Windows ME, so
> > Windows ME may still think it has the oplock.  20 minutes later, after
> > viewing the file a couple more times in ME, and after editing it
> > locally, there are no further messages with "oplock" or "linux" in the
> > Samba log.
> 
> This makes sense as Windows still thinks it has the oplock.
> 
> > It looks like either:
> > 
> >   (1) Linux leases aren't working: they're not signalling Samba.
> > 
> >   (2) or, Samba is not interfacing with Linux leases properly, so
> >       is missing the signal.
> 
> Either is possible.  Is this a standard kernel.org 2.6.8 kernel and
> a standard samba 3.0.6 - or are these distribution versions?

Gentoo versions.

Kernel is gentoo-dev-sources-2.6.8, which is Linux 2.6.8 plus a few small
patches that don't touch locks.c or anything that looks relevant.

Samba is samba-3.0.6-r3, again none of the small patches looks relevant.

-- Jamie

Re: file leases

[Jeremy Allison]

On Wed, Sep 22, 2004 at 12:32:32PM +0100, Jamie Lokier wrote:

> [2004/09/22 11:39:10, 3] smbd/oplock_linux.c:linux_oplock_receive_message(149)
>   linux_oplock_receive_message: kernel oplock break request received for dev = 303, inode = 1325168 fd = 28, fileid = 6

This shows that smbd is being signalled.

> Perhaps more worryingly, there is no communication with Windows ME, so
> Windows ME may still think it has the oplock.  20 minutes later, after
> viewing the file a couple more times in ME, and after editing it
> locally, there are no further messages with "oplock" or "linux" in the
> Samba log.

I need to see the complete debug level 10 log. Once smbd has received 
the message it should send the "oplock break request" to the client.
It's possible that the client is not responding to that, but this can
be due to a multitude of client issues (especially with Win9x based
code).

Ensure you have disabled sendfile on the smbd side, some clients have
a problem with this.

Jeremy.

Re: file leases

[Jamie Lokier]

Jeremy Allison wrote:
> On Wed, Sep 22, 2004 at 12:32:32PM +0100, Jamie Lokier wrote:
> > [2004/09/22 11:39:10, 3] smbd/oplock_linux.c:linux_oplock_receive_message(149)
> >   linux_oplock_receive_message: kernel oplock break request received for dev = 303, inode = 1325168 fd = 28, fileid = 6
> 
> This shows that smbd is being signalled.

No, that's an occasion where there wasn't a stall.  Check the dates:

jamie@kushida ~/x $ date; cp -p 3rd.jpg 4th.jpg; date
Wed Sep 22 11:50:49 BST 2004
Wed Sep 22 11:51:34 BST 2004

The "grep linux_ /var/log/samba/log.smbd" is done _after_ that.
This is the last message containing linux_ in the log:

[2004/09/22 11:50:40, 3] smbd/oplock_linux.c:linux_set_kernel_oplock(185)
  linux_set_kernel_oplock: got kernel oplock on file x/3rd.jpg, dev = 303, inode = 1325168, file_id = 7

> > Perhaps more worryingly, there is no communication with Windows ME, so
> > Windows ME may still think it has the oplock.  20 minutes later, after
> > viewing the file a couple more times in ME, and after editing it
> > locally, there are no further messages with "oplock" or "linux" in the
> > Samba log.
> 
> I need to see the complete debug level 10 log. Once smbd has received 
> the message it should send the "oplock break request" to the client.
> It's possible that the client is not responding to that, but this can
> be due to a multitude of client issues (especially with Win9x based
> code).

No, Samba is not receiving the lease message at the times when
operations stall for 45 seconds (or it isn't logging it).

The line you picked out was from a time when it didn't stall - that's
showing us that these messages are delivered sometimes, but not always.

I know you want to see a detailed trace, so I prepared one.

These are the commands I did on the server.  Notice how some of them
stall and some don't.  I've inserted the places where I opened a file
in Internet Explorer and then closed it again, on the ME client:

[ Opened test.jpg on client; then closed it ]
jamie@kushida ~/x $ date; cp -p test.jpg 2nd.jpg; date
Wed Sep 22 22:29:53 BST 2004
Wed Sep 22 22:30:38 BST 2004
[ Opened test.jpg on client; then closed it ]
jamie@kushida ~/x $ date; cp -p test.jpg 2nd.jpg; date
Wed Sep 22 22:30:50 BST 2004
Wed Sep 22 22:30:50 BST 2004
[ Opened test.jpg on client; then closed it ]
jamie@kushida ~/x $ date; cp -p test.jpg 2nd.jpg; date
Wed Sep 22 22:31:01 BST 2004
Wed Sep 22 22:31:01 BST 2004
[ Opened 2nd.jpg on client; then closed it ]
jamie@kushida ~/x $ date; cp -p 2nd.jpg 3rd.jpg; date
Wed Sep 22 22:31:23 BST 2004
Wed Sep 22 22:32:08 BST 2004
[ Didn't do anything on the client ]
jamie@kushida ~/x $ date; cp -p 2nd.jpg 3rd.jpg; date
Wed Sep 22 22:32:22 BST 2004
Wed Sep 22 22:32:22 BST 2004
[ Opened 2nd.jpg on client; then closed it ]
jamie@kushida ~/x $ date; cp -p 2nd.jpg 3rd.jpg; date
Wed Sep 22 22:32:34 BST 2004
Wed Sep 22 22:32:34 BST 2004
[ Opened 2nd.jpg on client; then closed it ]
jamie@kushida ~/x $ date; cp -p 2nd.jpg 3rd.jpg; date
Wed Sep 22 22:32:45 BST 2004
Wed Sep 22 22:32:45 BST 2004

There are four log files, two of which are too large to post to the
list so I'll send them to you (Jeremy) directly.  (Anyone else who
wants these files, just ask).

    1. log.smbd starting before the first action above, ending after
       the last, at log level 10
2,3,4. strace of the three smbd processes.

-- Jamie

Re: file leases

[William A.(Andy) Adamson]

ah. i have performed similar tests with nfsd and delegations. i'll open a file 
for READ locally on a version 4 nfsd exported portion of the file system, and 
request a lease via fcntl_setlease() setting up a signal handler to handle the 
break_lease callback. then, i'll open the same file for WRITE from an nfs 
version 4 client. the lease on the local file is broken within 1-2 seconds, 
and then nfs client WRITE is serviced. if you want, i can send the source to 
the test programs to set a lease.

-->Andy


--------jamie@shareable.org said -----------
Eh?  There's only one client, and it's idle.  Presumably it's holding
an oplock.

When I want to edit a file directly on the server, opening for write
stalls for a long time.  Presumably the lease causes Samba to send a
break_lease to the _one_ client.  Being an idle client that's not
holding the file open, it should respond immediately to break_lease,
and then Samba should let me open the file locally.

That's how leases are supposed to work, but something in that whole
arrangement isn't working properly.  Samba is 3.0.6-r3, kernel is
2.6.8 (both gentoo versions).