Temporary hangs when using locking with apache+nfsv4

Temporary hangs when using locking with apache+nfsv4

[Dennis Jacobfeuerborn]

Hi,
I'm experimenting with using NFSv4 as storage for web servers and while 
regular file access seems to work fine as soon as I bring flock() into 
play things become more problematic.
I've create a tiny test php script that basically opens a file, locks it 
using flock(), writes that fact into a log file (on a local filesystem), 
performs a usleep(1000), writes into the log that it is about to unlock 
the file and finally unlocks it.
I invoke that script using ab with a concurrency of 20 for a few 
thousand requests.

The result is that while 99% of the request respond quickly a few 
request seem to hang for up to 30 seconds. According to the log file 
they must eventually succeed since I see all expected entries and the 
locking seems to work as well since all entries are in the expected order.

Is it expected that these long delays happen? When I comment the locking 
function out these hangs disappear.
Are there some knobs to tune NFS and make it behave better in these 
situations?

Regards,
   Dennis

Re: Temporary hangs when using locking with apache+nfsv4

[Jeff Layton]

On Mon, 03 Mar 2014 06:47:52 +0100
Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:

> Hi,
> I'm experimenting with using NFSv4 as storage for web servers and while 
> regular file access seems to work fine as soon as I bring flock() into 
> play things become more problematic.
> I've create a tiny test php script that basically opens a file, locks it 
> using flock(), writes that fact into a log file (on a local filesystem), 
> performs a usleep(1000), writes into the log that it is about to unlock 
> the file and finally unlocks it.
> I invoke that script using ab with a concurrency of 20 for a few 
> thousand requests.
> 

Is all the activity from a single client, or are multiple clients
contending for the lock?

> The result is that while 99% of the request respond quickly a few 
> request seem to hang for up to 30 seconds. According to the log file 
> they must eventually succeed since I see all expected entries and the 
> locking seems to work as well since all entries are in the expected order.
> 
> Is it expected that these long delays happen? When I comment the locking 
> function out these hangs disappear.
> Are there some knobs to tune NFS and make it behave better in these 
> situations?
> 

NFSv4 locking is inherently unfair. If you're doing a blocking lock,
then the client is expected to poll for it. So, long delays are
possible if you just happen to be unlucky and keep missing the lock.

There's no knob to tune, but there probably is room for improvement in
this code. In principle we could try to be more aggressive about
getting the lock by trying to wake up one or more blocked tasks whenever
a lock is released. You might still end up with delays, but it could
help improve responsiveness.

-- 
Jeff Layton <jlayton@redhat.com>

Re: Temporary hangs when using locking with apache+nfsv4

[Trond Myklebust]

On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:

> On Mon, 03 Mar 2014 06:47:52 +0100
> Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> 
>> Hi,
>> I'm experimenting with using NFSv4 as storage for web servers and while 
>> regular file access seems to work fine as soon as I bring flock() into 
>> play things become more problematic.
>> I've create a tiny test php script that basically opens a file, locks it 
>> using flock(), writes that fact into a log file (on a local filesystem), 
>> performs a usleep(1000), writes into the log that it is about to unlock 
>> the file and finally unlocks it.
>> I invoke that script using ab with a concurrency of 20 for a few 
>> thousand requests.
>> 
> 
> Is all the activity from a single client, or are multiple clients
> contending for the lock?
> 
>> The result is that while 99% of the request respond quickly a few 
>> request seem to hang for up to 30 seconds. According to the log file 
>> they must eventually succeed since I see all expected entries and the 
>> locking seems to work as well since all entries are in the expected order.
>> 
>> Is it expected that these long delays happen? When I comment the locking 
>> function out these hangs disappear.
>> Are there some knobs to tune NFS and make it behave better in these 
>> situations?
>> 
> 
> NFSv4 locking is inherently unfair. If you're doing a blocking lock,
> then the client is expected to poll for it. So, long delays are
> possible if you just happen to be unlucky and keep missing the lock.
> 
> There's no knob to tune, but there probably is room for improvement in
> this code. In principle we could try to be more aggressive about
> getting the lock by trying to wake up one or more blocked tasks whenever
> a lock is released. You might still end up with delays, but it could
> help improve responsiveness.

…or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.

_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@primarydata.com

Re: Temporary hangs when using locking with apache+nfsv4

[Jeff Layton]

On Mon, 3 Mar 2014 10:46:37 -0500
Trond Myklebust <trond.myklebust@primarydata.com> wrote:

> 
> On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:
> 
> > On Mon, 03 Mar 2014 06:47:52 +0100
> > Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> > 
> >> Hi,
> >> I'm experimenting with using NFSv4 as storage for web servers and while 
> >> regular file access seems to work fine as soon as I bring flock() into 
> >> play things become more problematic.
> >> I've create a tiny test php script that basically opens a file, locks it 
> >> using flock(), writes that fact into a log file (on a local filesystem), 
> >> performs a usleep(1000), writes into the log that it is about to unlock 
> >> the file and finally unlocks it.
> >> I invoke that script using ab with a concurrency of 20 for a few 
> >> thousand requests.
> >> 
> > 
> > Is all the activity from a single client, or are multiple clients
> > contending for the lock?
> > 
> >> The result is that while 99% of the request respond quickly a few 
> >> request seem to hang for up to 30 seconds. According to the log file 
> >> they must eventually succeed since I see all expected entries and the 
> >> locking seems to work as well since all entries are in the expected order.
> >> 
> >> Is it expected that these long delays happen? When I comment the locking 
> >> function out these hangs disappear.
> >> Are there some knobs to tune NFS and make it behave better in these 
> >> situations?
> >> 
> > 
> > NFSv4 locking is inherently unfair. If you're doing a blocking lock,
> > then the client is expected to poll for it. So, long delays are
> > possible if you just happen to be unlucky and keep missing the lock.
> > 
> > There's no knob to tune, but there probably is room for improvement in
> > this code. In principle we could try to be more aggressive about
> > getting the lock by trying to wake up one or more blocked tasks whenever
> > a lock is released. You might still end up with delays, but it could
> > help improve responsiveness.
> 
> …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.

I had forgotten about those. I wonder what servers actually implement
them? I don't think Linux' knfsd does yet.

I wasn't really suggesting more aggressive polling. The timer semantics
seem fine as they are, but we could short circuit it when we know that
a lock on the inode has just become free.

Maybe we could share the sillyrename waitqueue, and have clients sleep
on that. When we go to send the LOCKU request, we'd wake up the queue.

It's not any more fair, but could improve latency in some cases.

-- 
Jeff Layton <jlayton@redhat.com>

Re: Temporary hangs when using locking with apache+nfsv4

[Trond Myklebust]

On Mar 3, 2014, at 11:41, Jeff Layton <jlayton@redhat.com> wrote:

> On Mon, 3 Mar 2014 10:46:37 -0500
> Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> 
>> 
>> On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:
>> 
>>> On Mon, 03 Mar 2014 06:47:52 +0100
>>> Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
>>> 
>>>> Hi,
>>>> I'm experimenting with using NFSv4 as storage for web servers and while 
>>>> regular file access seems to work fine as soon as I bring flock() into 
>>>> play things become more problematic.
>>>> I've create a tiny test php script that basically opens a file, locks it 
>>>> using flock(), writes that fact into a log file (on a local filesystem), 
>>>> performs a usleep(1000), writes into the log that it is about to unlock 
>>>> the file and finally unlocks it.
>>>> I invoke that script using ab with a concurrency of 20 for a few 
>>>> thousand requests.
>>>> 
>>> 
>>> Is all the activity from a single client, or are multiple clients
>>> contending for the lock?
>>> 
>>>> The result is that while 99% of the request respond quickly a few 
>>>> request seem to hang for up to 30 seconds. According to the log file 
>>>> they must eventually succeed since I see all expected entries and the 
>>>> locking seems to work as well since all entries are in the expected order.
>>>> 
>>>> Is it expected that these long delays happen? When I comment the locking 
>>>> function out these hangs disappear.
>>>> Are there some knobs to tune NFS and make it behave better in these 
>>>> situations?
>>>> 
>>> 
>>> NFSv4 locking is inherently unfair. If you're doing a blocking lock,
>>> then the client is expected to poll for it. So, long delays are
>>> possible if you just happen to be unlucky and keep missing the lock.
>>> 
>>> There's no knob to tune, but there probably is room for improvement in
>>> this code. In principle we could try to be more aggressive about
>>> getting the lock by trying to wake up one or more blocked tasks whenever
>>> a lock is released. You might still end up with delays, but it could
>>> help improve responsiveness.
>> 
>> …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.
> 
> I had forgotten about those. I wonder what servers actually implement
> them? I don't think Linux' knfsd does yet.
> 
> I wasn't really suggesting more aggressive polling. The timer semantics
> seem fine as they are, but we could short circuit it when we know that
> a lock on the inode has just become free.

How do we “know” that the lock is free? We already track all the locks that our client holds, and wait for those to be released. I can’t see what else there is to do.

_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@primarydata.com

Re: Temporary hangs when using locking with apache+nfsv4

[Jeff Layton]

On Mon, 3 Mar 2014 13:22:29 -0500
Trond Myklebust <trond.myklebust@primarydata.com> wrote:

> 
> On Mar 3, 2014, at 11:41, Jeff Layton <jlayton@redhat.com> wrote:
> 
> > On Mon, 3 Mar 2014 10:46:37 -0500
> > Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> > 
> >> 
> >> On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:
> >> 
> >>> On Mon, 03 Mar 2014 06:47:52 +0100
> >>> Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> >>> 
> >>>> Hi,
> >>>> I'm experimenting with using NFSv4 as storage for web servers and while 
> >>>> regular file access seems to work fine as soon as I bring flock() into 
> >>>> play things become more problematic.
> >>>> I've create a tiny test php script that basically opens a file, locks it 
> >>>> using flock(), writes that fact into a log file (on a local filesystem), 
> >>>> performs a usleep(1000), writes into the log that it is about to unlock 
> >>>> the file and finally unlocks it.
> >>>> I invoke that script using ab with a concurrency of 20 for a few 
> >>>> thousand requests.
> >>>> 
> >>> 
> >>> Is all the activity from a single client, or are multiple clients
> >>> contending for the lock?
> >>> 
> >>>> The result is that while 99% of the request respond quickly a few 
> >>>> request seem to hang for up to 30 seconds. According to the log file 
> >>>> they must eventually succeed since I see all expected entries and the 
> >>>> locking seems to work as well since all entries are in the expected order.
> >>>> 
> >>>> Is it expected that these long delays happen? When I comment the locking 
> >>>> function out these hangs disappear.
> >>>> Are there some knobs to tune NFS and make it behave better in these 
> >>>> situations?
> >>>> 
> >>> 
> >>> NFSv4 locking is inherently unfair. If you're doing a blocking lock,
> >>> then the client is expected to poll for it. So, long delays are
> >>> possible if you just happen to be unlucky and keep missing the lock.
> >>> 
> >>> There's no knob to tune, but there probably is room for improvement in
> >>> this code. In principle we could try to be more aggressive about
> >>> getting the lock by trying to wake up one or more blocked tasks whenever
> >>> a lock is released. You might still end up with delays, but it could
> >>> help improve responsiveness.
> >> 
> >> …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.
> > 
> > I had forgotten about those. I wonder what servers actually implement
> > them? I don't think Linux' knfsd does yet.
> > 
> > I wasn't really suggesting more aggressive polling. The timer semantics
> > seem fine as they are, but we could short circuit it when we know that
> > a lock on the inode has just become free.
> 
> How do we “know” that the lock is free? We already track all the locks that our client holds, and wait for those to be released. I can’t see what else there is to do.
> 

Right, we do that, but tasks that are polling for the lock don't get
woken up when a task releases a lock. They currently just wait until
the timeout occurs and then attempt to acquire the lock. The pessimal
case is that:

- try to acquire the lock and be denied
- task goes to sleep for 30s
- just after that, another task releases the lock

The first task will wait for 30s before retrying when it could have
gotten the lock soon afterward.

The idea would be to go ahead and wake up all the blocked waiters on an
inode when a task releases a lock. They'd then just re-attempt
acquiring the lock immediately instead of waiting on the timeout.

On a highly contended lock, most of the waiters would just go back to
sleep after being denied again, but one might end up getting the lock
and keeping things moving.

We could also try to be clever and only wake up tasks that are blocked
on the range being released, but in Dennis' case, he's using flock()
so that wouldn't really buy him anything.

-- 
Jeff Layton <jlayton@redhat.com>

Re: Temporary hangs when using locking with apache+nfsv4

[Trond Myklebust]

On Mar 3, 2014, at 13:34, Jeff Layton <jlayton@redhat.com> wrote:

> On Mon, 3 Mar 2014 13:22:29 -0500
> Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> 
>> 
>> On Mar 3, 2014, at 11:41, Jeff Layton <jlayton@redhat.com> wrote:
>> 
>>> On Mon, 3 Mar 2014 10:46:37 -0500
>>> Trond Myklebust <trond.myklebust@primarydata.com> wrote:
>>> 
>>>> 
>>>> On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:
>>>> 
>>>>> On Mon, 03 Mar 2014 06:47:52 +0100
>>>>> Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> I'm experimenting with using NFSv4 as storage for web servers and while 
>>>>>> regular file access seems to work fine as soon as I bring flock() into 
>>>>>> play things become more problematic.
>>>>>> I've create a tiny test php script that basically opens a file, locks it 
>>>>>> using flock(), writes that fact into a log file (on a local filesystem), 
>>>>>> performs a usleep(1000), writes into the log that it is about to unlock 
>>>>>> the file and finally unlocks it.
>>>>>> I invoke that script using ab with a concurrency of 20 for a few 
>>>>>> thousand requests.
>>>>>> 
>>>>> 
>>>>> Is all the activity from a single client, or are multiple clients
>>>>> contending for the lock?
>>>>> 
>>>>>> The result is that while 99% of the request respond quickly a few 
>>>>>> request seem to hang for up to 30 seconds. According to the log file 
>>>>>> they must eventually succeed since I see all expected entries and the 
>>>>>> locking seems to work as well since all entries are in the expected order.
>>>>>> 
>>>>>> Is it expected that these long delays happen? When I comment the locking 
>>>>>> function out these hangs disappear.
>>>>>> Are there some knobs to tune NFS and make it behave better in these 
>>>>>> situations?
>>>>>> 
>>>>> 
>>>>> NFSv4 locking is inherently unfair. If you're doing a blocking lock,
>>>>> then the client is expected to poll for it. So, long delays are
>>>>> possible if you just happen to be unlucky and keep missing the lock.
>>>>> 
>>>>> There's no knob to tune, but there probably is room for improvement in
>>>>> this code. In principle we could try to be more aggressive about
>>>>> getting the lock by trying to wake up one or more blocked tasks whenever
>>>>> a lock is released. You might still end up with delays, but it could
>>>>> help improve responsiveness.
>>>> 
>>>> …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.
>>> 
>>> I had forgotten about those. I wonder what servers actually implement
>>> them? I don't think Linux' knfsd does yet.
>>> 
>>> I wasn't really suggesting more aggressive polling. The timer semantics
>>> seem fine as they are, but we could short circuit it when we know that
>>> a lock on the inode has just become free.
>> 
>> How do we “know” that the lock is free? We already track all the locks that our client holds, and wait for those to be released. I can’t see what else there is to do.
>> 
> 
> Right, we do that, but tasks that are polling for the lock don't get
> woken up when a task releases a lock. They currently just wait until
> the timeout occurs and then attempt to acquire the lock. The pessimal
> case is that:
> 
> - try to acquire the lock and be denied
> - task goes to sleep for 30s
> - just after that, another task releases the lock
> 
> The first task will wait for 30s before retrying when it could have
> gotten the lock soon afterward.
> 
> The idea would be to go ahead and wake up all the blocked waiters on an
> inode when a task releases a lock. They'd then just re-attempt
> acquiring the lock immediately instead of waiting on the timeout.
> 
> On a highly contended lock, most of the waiters would just go back to
> sleep after being denied again, but one might end up getting the lock
> and keeping things moving.
> 
> We could also try to be clever and only wake up tasks that are blocked
> on the range being released, but in Dennis' case, he's using flock()
> so that wouldn't really buy him anything.

How about just resetting the backoff timer when the call to do_vfs_lock() sleeps due to a client-internal lock contention?

_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@primarydata.com

Re: Temporary hangs when using locking with apache+nfsv4

[Jeff Layton]

On Mon, 3 Mar 2014 14:02:29 -0500
Trond Myklebust <trond.myklebust@primarydata.com> wrote:

> 
> On Mar 3, 2014, at 13:34, Jeff Layton <jlayton@redhat.com> wrote:
> 
> > On Mon, 3 Mar 2014 13:22:29 -0500
> > Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> > 
> >> 
> >> On Mar 3, 2014, at 11:41, Jeff Layton <jlayton@redhat.com> wrote:
> >> 
> >>> On Mon, 3 Mar 2014 10:46:37 -0500
> >>> Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> >>> 
> >>>> 
> >>>> On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:
> >>>> 
> >>>>> On Mon, 03 Mar 2014 06:47:52 +0100
> >>>>> Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> >>>>> 
> >>>>>> Hi,
> >>>>>> I'm experimenting with using NFSv4 as storage for web servers and while 
> >>>>>> regular file access seems to work fine as soon as I bring flock() into 
> >>>>>> play things become more problematic.
> >>>>>> I've create a tiny test php script that basically opens a file, locks it 
> >>>>>> using flock(), writes that fact into a log file (on a local filesystem), 
> >>>>>> performs a usleep(1000), writes into the log that it is about to unlock 
> >>>>>> the file and finally unlocks it.
> >>>>>> I invoke that script using ab with a concurrency of 20 for a few 
> >>>>>> thousand requests.
> >>>>>> 
> >>>>> 
> >>>>> Is all the activity from a single client, or are multiple clients
> >>>>> contending for the lock?
> >>>>> 
> >>>>>> The result is that while 99% of the request respond quickly a few 
> >>>>>> request seem to hang for up to 30 seconds. According to the log file 
> >>>>>> they must eventually succeed since I see all expected entries and the 
> >>>>>> locking seems to work as well since all entries are in the expected order.
> >>>>>> 
> >>>>>> Is it expected that these long delays happen? When I comment the locking 
> >>>>>> function out these hangs disappear.
> >>>>>> Are there some knobs to tune NFS and make it behave better in these 
> >>>>>> situations?
> >>>>>> 
> >>>>> 
> >>>>> NFSv4 locking is inherently unfair. If you're doing a blocking lock,
> >>>>> then the client is expected to poll for it. So, long delays are
> >>>>> possible if you just happen to be unlucky and keep missing the lock.
> >>>>> 
> >>>>> There's no knob to tune, but there probably is room for improvement in
> >>>>> this code. In principle we could try to be more aggressive about
> >>>>> getting the lock by trying to wake up one or more blocked tasks whenever
> >>>>> a lock is released. You might still end up with delays, but it could
> >>>>> help improve responsiveness.
> >>>> 
> >>>> …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.
> >>> 
> >>> I had forgotten about those. I wonder what servers actually implement
> >>> them? I don't think Linux' knfsd does yet.
> >>> 
> >>> I wasn't really suggesting more aggressive polling. The timer semantics
> >>> seem fine as they are, but we could short circuit it when we know that
> >>> a lock on the inode has just become free.
> >> 
> >> How do we “know” that the lock is free? We already track all the locks that our client holds, and wait for those to be released. I can’t see what else there is to do.
> >> 
> > 
> > Right, we do that, but tasks that are polling for the lock don't get
> > woken up when a task releases a lock. They currently just wait until
> > the timeout occurs and then attempt to acquire the lock. The pessimal
> > case is that:
> > 
> > - try to acquire the lock and be denied
> > - task goes to sleep for 30s
> > - just after that, another task releases the lock
> > 
> > The first task will wait for 30s before retrying when it could have
> > gotten the lock soon afterward.
> > 
> > The idea would be to go ahead and wake up all the blocked waiters on an
> > inode when a task releases a lock. They'd then just re-attempt
> > acquiring the lock immediately instead of waiting on the timeout.
> > 
> > On a highly contended lock, most of the waiters would just go back to
> > sleep after being denied again, but one might end up getting the lock
> > and keeping things moving.
> > 
> > We could also try to be clever and only wake up tasks that are blocked
> > on the range being released, but in Dennis' case, he's using flock()
> > so that wouldn't really buy him anything.
> 
> How about just resetting the backoff timer when the call to do_vfs_lock() sleeps due to a client-internal lock contention?
> 

Hmm, maybe. Looking at how this works in _nfs4_proc_setlk...

Assuming this is a blocking lock request, we first do an FL_ACCESS
request that blocks. Once that comes free, we then issue the LOCK
request to server and then set the vfs-layer lock if we get it.

We don't currently have a way to tell whether the initial FL_ACCESS
request blocked before returning or not. I suppose we could try to
plumb that into the vfs-layer locking code. That might not be too
hard...

-- 
Jeff Layton <jlayton@redhat.com>

Re: Temporary hangs when using locking with apache+nfsv4

[J. Bruce Fields]

On Mon, Mar 03, 2014 at 11:41:19AM -0500, Jeff Layton wrote:
> On Mon, 3 Mar 2014 10:46:37 -0500
> Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> 
> > 
> > On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:
> > 
> > > On Mon, 03 Mar 2014 06:47:52 +0100
> > > Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> > > 
> > >> Hi,
> > >> I'm experimenting with using NFSv4 as storage for web servers and while 
> > >> regular file access seems to work fine as soon as I bring flock() into 
> > >> play things become more problematic.
> > >> I've create a tiny test php script that basically opens a file, locks it 
> > >> using flock(), writes that fact into a log file (on a local filesystem), 
> > >> performs a usleep(1000), writes into the log that it is about to unlock 
> > >> the file and finally unlocks it.
> > >> I invoke that script using ab with a concurrency of 20 for a few 
> > >> thousand requests.
> > >> 
> > > 
> > > Is all the activity from a single client, or are multiple clients
> > > contending for the lock?
> > > 
> > >> The result is that while 99% of the request respond quickly a few 
> > >> request seem to hang for up to 30 seconds. According to the log file 
> > >> they must eventually succeed since I see all expected entries and the 
> > >> locking seems to work as well since all entries are in the expected order.
> > >> 
> > >> Is it expected that these long delays happen? When I comment the locking 
> > >> function out these hangs disappear.
> > >> Are there some knobs to tune NFS and make it behave better in these 
> > >> situations?
> > >> 
> > > 
> > > NFSv4 locking is inherently unfair. If you're doing a blocking lock,
> > > then the client is expected to poll for it. So, long delays are
> > > possible if you just happen to be unlucky and keep missing the lock.
> > > 
> > > There's no knob to tune, but there probably is room for improvement in
> > > this code. In principle we could try to be more aggressive about
> > > getting the lock by trying to wake up one or more blocked tasks whenever
> > > a lock is released. You might still end up with delays, but it could
> > > help improve responsiveness.
> > 
> > …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.
> 
> I had forgotten about those. I wonder what servers actually implement
> them? I don't think Linux' knfsd does yet.

No.  How I'd imagined it would work:

	- on a failed blocking lock request, insert a waiter.
	- when the lock the waiter is blocking on is released or
	  downgraded, apply the waiting lock as a "provisional" lock:
	  add it to the i_flock list, but *don't* allow it to downgrade
	  or merge with any existing locks.  Then send the callback.
	- when the client resends the lock request, finish applying the
	  lock.  This is when we downgrade, merge, or split as
	  necessary.
	- Alternatively, if some timeout passes without the client
	  requesting the lock again, give up and remove the
	  "provisional" lock.

Then we need to implement the client side too.  And there are some more
(optional) suggestions in 9.6.

--b.

> I wasn't really suggesting more aggressive polling. The timer semantics
> seem fine as they are, but we could short circuit it when we know that
> a lock on the inode has just become free.
> 
> Maybe we could share the sillyrename waitqueue, and have clients sleep
> on that. When we go to send the LOCKU request, we'd wake up the queue.
> 
> It's not any more fair, but could improve latency in some cases.
> 
> -- 
> Jeff Layton <jlayton@redhat.com>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Temporary hangs when using locking with apache+nfsv4

[Jeff Layton]

On Mon, 3 Mar 2014 15:41:54 -0500
"J. Bruce Fields" <bfields@fieldses.org> wrote:

> On Mon, Mar 03, 2014 at 11:41:19AM -0500, Jeff Layton wrote:
> > On Mon, 3 Mar 2014 10:46:37 -0500
> > Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> > 
> > > 
> > > On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:
> > > 
> > > > On Mon, 03 Mar 2014 06:47:52 +0100
> > > > Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> > > > 
> > > >> Hi,
> > > >> I'm experimenting with using NFSv4 as storage for web servers and while 
> > > >> regular file access seems to work fine as soon as I bring flock() into 
> > > >> play things become more problematic.
> > > >> I've create a tiny test php script that basically opens a file, locks it 
> > > >> using flock(), writes that fact into a log file (on a local filesystem), 
> > > >> performs a usleep(1000), writes into the log that it is about to unlock 
> > > >> the file and finally unlocks it.
> > > >> I invoke that script using ab with a concurrency of 20 for a few 
> > > >> thousand requests.
> > > >> 
> > > > 
> > > > Is all the activity from a single client, or are multiple clients
> > > > contending for the lock?
> > > > 
> > > >> The result is that while 99% of the request respond quickly a few 
> > > >> request seem to hang for up to 30 seconds. According to the log file 
> > > >> they must eventually succeed since I see all expected entries and the 
> > > >> locking seems to work as well since all entries are in the expected order.
> > > >> 
> > > >> Is it expected that these long delays happen? When I comment the locking 
> > > >> function out these hangs disappear.
> > > >> Are there some knobs to tune NFS and make it behave better in these 
> > > >> situations?
> > > >> 
> > > > 
> > > > NFSv4 locking is inherently unfair. If you're doing a blocking lock,
> > > > then the client is expected to poll for it. So, long delays are
> > > > possible if you just happen to be unlucky and keep missing the lock.
> > > > 
> > > > There's no knob to tune, but there probably is room for improvement in
> > > > this code. In principle we could try to be more aggressive about
> > > > getting the lock by trying to wake up one or more blocked tasks whenever
> > > > a lock is released. You might still end up with delays, but it could
> > > > help improve responsiveness.
> > > 
> > > …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.
> > 
> > I had forgotten about those. I wonder what servers actually implement
> > them? I don't think Linux' knfsd does yet.
> 
> No.  How I'd imagined it would work:
> 
> 	- on a failed blocking lock request, insert a waiter.
> 	- when the lock the waiter is blocking on is released or
> 	  downgraded, apply the waiting lock as a "provisional" lock:
> 	  add it to the i_flock list, but *don't* allow it to downgrade
> 	  or merge with any existing locks.  Then send the callback.
> 	- when the client resends the lock request, finish applying the
> 	  lock.  This is when we downgrade, merge, or split as
> 	  necessary.
> 	- Alternatively, if some timeout passes without the client
> 	  requesting the lock again, give up and remove the
> 	  "provisional" lock.
> 

Do we really need to do that?

RFC5667 seems to indicate that the server isn't required to hold the
lock for the client when it sends the callback.

As a first step, we could just add the callbacks and not try to hold
the lock for the client. That wouldn't be too hard to do -- maybe just
add a blocking FL_ACCESS request to the i_flock list and then issue
a CB_NOTIFY_LOCK when that returns.

> Then we need to implement the client side too.  And there are some more
> (optional) suggestions in 9.6.
> 
> --b.
> 
> > I wasn't really suggesting more aggressive polling. The timer semantics
> > seem fine as they are, but we could short circuit it when we know that
> > a lock on the inode has just become free.
> > 
> > Maybe we could share the sillyrename waitqueue, and have clients sleep
> > on that. When we go to send the LOCKU request, we'd wake up the queue.
> > 
> > It's not any more fair, but could improve latency in some cases.
> > 
> > -- 
> > Jeff Layton <jlayton@redhat.com>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Jeff Layton <jlayton@redhat.com>

Re: Temporary hangs when using locking with apache+nfsv4

[J. Bruce Fields]

On Mon, Mar 03, 2014 at 05:29:21PM -0500, Jeff Layton wrote:
> On Mon, 3 Mar 2014 15:41:54 -0500
> "J. Bruce Fields" <bfields@fieldses.org> wrote:
> 
> > On Mon, Mar 03, 2014 at 11:41:19AM -0500, Jeff Layton wrote:
> > > On Mon, 3 Mar 2014 10:46:37 -0500
> > > Trond Myklebust <trond.myklebust@primarydata.com> wrote:
> > > 
> > > > 
> > > > On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@redhat.com> wrote:
> > > > 
> > > > > On Mon, 03 Mar 2014 06:47:52 +0100
> > > > > Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> > > > > 
> > > > >> Hi,
> > > > >> I'm experimenting with using NFSv4 as storage for web servers and while 
> > > > >> regular file access seems to work fine as soon as I bring flock() into 
> > > > >> play things become more problematic.
> > > > >> I've create a tiny test php script that basically opens a file, locks it 
> > > > >> using flock(), writes that fact into a log file (on a local filesystem), 
> > > > >> performs a usleep(1000), writes into the log that it is about to unlock 
> > > > >> the file and finally unlocks it.
> > > > >> I invoke that script using ab with a concurrency of 20 for a few 
> > > > >> thousand requests.
> > > > >> 
> > > > > 
> > > > > Is all the activity from a single client, or are multiple clients
> > > > > contending for the lock?
> > > > > 
> > > > >> The result is that while 99% of the request respond quickly a few 
> > > > >> request seem to hang for up to 30 seconds. According to the log file 
> > > > >> they must eventually succeed since I see all expected entries and the 
> > > > >> locking seems to work as well since all entries are in the expected order.
> > > > >> 
> > > > >> Is it expected that these long delays happen? When I comment the locking 
> > > > >> function out these hangs disappear.
> > > > >> Are there some knobs to tune NFS and make it behave better in these 
> > > > >> situations?
> > > > >> 
> > > > > 
> > > > > NFSv4 locking is inherently unfair. If you're doing a blocking lock,
> > > > > then the client is expected to poll for it. So, long delays are
> > > > > possible if you just happen to be unlucky and keep missing the lock.
> > > > > 
> > > > > There's no knob to tune, but there probably is room for improvement in
> > > > > this code. In principle we could try to be more aggressive about
> > > > > getting the lock by trying to wake up one or more blocked tasks whenever
> > > > > a lock is released. You might still end up with delays, but it could
> > > > > help improve responsiveness.
> > > > 
> > > > …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling.
> > > 
> > > I had forgotten about those. I wonder what servers actually implement
> > > them? I don't think Linux' knfsd does yet.
> > 
> > No.  How I'd imagined it would work:
> > 
> > 	- on a failed blocking lock request, insert a waiter.
> > 	- when the lock the waiter is blocking on is released or
> > 	  downgraded, apply the waiting lock as a "provisional" lock:
> > 	  add it to the i_flock list, but *don't* allow it to downgrade
> > 	  or merge with any existing locks.  Then send the callback.
> > 	- when the client resends the lock request, finish applying the
> > 	  lock.  This is when we downgrade, merge, or split as
> > 	  necessary.
> > 	- Alternatively, if some timeout passes without the client
> > 	  requesting the lock again, give up and remove the
> > 	  "provisional" lock.
> > 
> 
> Do we really need to do that?
> 
> RFC5667 seems to indicate that the server isn't required to hold the
> lock for the client when it sends the callback.
> 
> As a first step, we could just add the callbacks and not try to hold
> the lock for the client. That wouldn't be too hard to do -- maybe just
> add a blocking FL_ACCESS request to the i_flock list and then issue
> a CB_NOTIFY_LOCK when that returns.

Yes, you're right, something like that is probably a better first step.

--b.

Re: Temporary hangs when using locking with apache+nfsv4

[Dennis Jacobfeuerborn]

On 03.03.2014 16:43, Jeff Layton wrote:
> On Mon, 03 Mar 2014 06:47:52 +0100
> Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
>
>> Hi,
>> I'm experimenting with using NFSv4 as storage for web servers and while
>> regular file access seems to work fine as soon as I bring flock() into
>> play things become more problematic.
>> I've create a tiny test php script that basically opens a file, locks it
>> using flock(), writes that fact into a log file (on a local filesystem),
>> performs a usleep(1000), writes into the log that it is about to unlock
>> the file and finally unlocks it.
>> I invoke that script using ab with a concurrency of 20 for a few
>> thousand requests.
>>
>
> Is all the activity from a single client, or are multiple clients
> contending for the lock?
>

"ab" is a benchmarking tool that simulates multiple clients using 
threads but I invoke only a single instance of it on a single system if 
that matters.

>> The result is that while 99% of the request respond quickly a few
>> request seem to hang for up to 30 seconds. According to the log file
>> they must eventually succeed since I see all expected entries and the
>> locking seems to work as well since all entries are in the expected order.
>>
>> Is it expected that these long delays happen? When I comment the locking
>> function out these hangs disappear.
>> Are there some knobs to tune NFS and make it behave better in these
>> situations?
>>
>
> NFSv4 locking is inherently unfair. If you're doing a blocking lock,
> then the client is expected to poll for it. So, long delays are
> possible if you just happen to be unlucky and keep missing the lock.

That's likely what is happening and I'm going to extend the test script 
with additional logging to verify this.

The script is also deliberately a bit more aggressive to test the 
behavior of the locking because I wanted to test the improved locking 
reliability of NFSv4 vs v3. The real-world test case is a CMS (Typo3) 
that serves pages from a cache but ises lock files when the cached 
version of that pages expires and has to be regenerated to prevent 
multiple processes re-generating the page at the same time.
So in the real-world case there will probably less contention and a few 
seconds between locking and unlocking. Also I have to check if the lock 
used by the CMS is blocking which seems unlikely since that would block 
all parallel request at least for the duration of the rendering of the page.

Regards,
   Dennis