Fcntl F_SETLKW hangs on Linux NFS mount

Fcntl F_SETLKW hangs on Linux NFS mount

[khowe]

[-- Attachment #1: Type: text/plain, Size: 4106 bytes --]

I'm running into an issue when using fcntl with the F_SETLKW command on
a Linux NFS client. In cases where I have many processes all trying to
get a write lock at once, the call to fcntl never returns, even though
the lock appears to have been granted successfully. It is not a
consistent error, but it is fairly repeatable (about 30% of the time).
I'm running on a RedHat client, though I'm not exactly sure of the
RedHat version. However, the kernel reported by uname is "Linux
prblnx-bo04 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:17:59 EDT 2005 i686
i686 i386 GNU/Linux", and it appears to be using v1.0.6 of nfs-utils.
The mounts are set via autofs, using automount v4.1.3-130.

My test program creates 16 process that all try to get a write lock on
the same file, and it will frequently fail such that one of the
processes appears to have been granted the lock but does not return from
the fcntl call (and the remaining processes are then also all stuck).
Listing open files and locks using "lsof -N" shows that one of the
processes has been granted the W lock, yet this same process does not
return from the fcntl call. I have toyed with the mounting options on
the client and I am seeing this behavior with NFS v2 and v3, with short
or long timeouts (from timeo=7 to timeo=600), with intr and with nointr,
and with r and wsize's ranging from 1024 to 32768. The only option I
haven't been able to mess with is using TCP instead of UDP, since this
particular RedHat build does not seem to allow the TCP option.

The NFS server for the mount is a vendor provided NAS device, and I'm
not sure what its OS or NFS server is. However, the fact that lsof shows
the lock has been granted seems to indicate the disconnect is somewhere
on the client side after getting the lock but before waking up fcntl. I
also run the exact some program on a Solaris host against the same mount
and have no issues at all, which also seems to rule out a server-side
issue.

The test program I'm using is a c app compiled with gcc v3.3.1. Any
thoughts? Thanks...

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/time.h>

extern int errno;

double now(void) {
  struct timeval tv;
  struct timezone tz;
  
  gettimeofday(&tv, &tz);
  
  return((double)tv.tv_sec + (double)tv.tv_usec/1000000);
}

int main(int argc, char **argv) {
  int i, j, id = 0;
  
  // create test lock file
  int lock_h = open(argv[1], O_CREAT | O_TRUNC | O_WRONLY, S_IRWXU);
  if (lock_h == -1) {
    printf("failed to create lock file: %s\n", strerror(errno));
    exit(2);
  }
  else {
    close(lock_h);
  }
  
  // fork some workers
  pid_t pids[4] = { 0, 0, 0, 0 };
  for (i = 3; i >= 0; i--) {
    pid_t pid = fork();
    if (pid == -1) {
      printf("failed to fork: %s\n", strerror(errno));
      exit(3);
    }
    if (pid == 0) {
      int pow = 1;
      for (j = 0; j < i; j++) {
        pow *= 2;
      }
      id += pow;
    }
    pids[i] = pid;
  }
  
  pid_t mypid = getpid();
  
  // open lock file for write and try to lock it
  int status;
  struct flock lock;
  lock.l_type = F_WRLCK;
  lock.l_whence = 0;
  lock.l_start = 0;
  lock.l_len = 0;
  lock_h = open(argv[1], O_WRONLY);
  if (lock_h == -1) {
    printf("failed to open lock file: %s\n", strerror(errno));
    exit(4);
  }
  printf("%15.5f %05d %s\n", now(), mypid, "trying lock");
  status = fcntl(lock_h, F_SETLKW, &lock);
  if (status == 0) {
    printf("%15.5f %05d %s\n", now(), mypid, "got lock");
    lock.l_type = F_UNLCK;
    fcntl(lock_h, F_SETLKW, &lock);
    printf("%15.5f %05d %s\n", now(), mypid, "freed lock");
  }
  else {
    printf("%15.5f %05d %s\n", now(), mypid, "failed lock");
    printf("%15.5f %05d %s\n", now(), mypid, strerror(errno));
  }
  close(lock_h);
  
  // cleanup and quit
  for (i = 0; i < 4; i++) {
    if (pids[i] != 0) {
      waitpid(pids[i], &status, 0);
    }
  }
  if (id == 0) {
    unlink(argv[1]);
  }
  
  return(0);
}

[-- Attachment #2: Type: text/html, Size: 9134 bytes --]

Re: Fcntl F_SETLKW hangs on Linux NFS mount

[Steve Dickson]

khowe@micron.com wrote:
> I'm running into an issue when using fcntl with the F_SETLKW command on 
> a Linux NFS client. In cases where I have many processes all trying to 
> get a write lock at once, the call to fcntl never returns, even though 
> the lock appears to have been granted successfully. It is not a 
> consistent error, but it is fairly repeatable (about 30% of the time). 
> I'm running on a RedHat client, though I'm not exactly sure of the 
> RedHat version. However, the kernel reported by uname is "Linux 
> prblnx-bo04 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:17:59 EDT 2005 i686 
> i686 i386 GNU/Linux", and it appears to be using v1.0.6 of nfs-utils. 
> The mounts are set via autofs, using automount v4.1.3-130.
> 
> My test program creates 16 process that all try to get a write lock on 
> the same file, and it will frequently fail such that one of the 
> processes appears to have been granted the lock but does not return from 
> the fcntl call (and the remaining processes are then also all stuck). 
> Listing open files and locks using "lsof -N" shows that one of the 
> processes has been granted the W lock, yet this same process does not 
> return from the fcntl call. I have toyed with the mounting options on 
> the client and I am seeing this behavior with NFS v2 and v3, with short 
> or long timeouts (from timeo=7 to timeo=600), with intr and with nointr, 
> and with r and wsize's ranging from 1024 to 32768. The only option I 
> haven't been able to mess with is using TCP instead of UDP, since this 
> particular RedHat build does not seem to allow the TCP option.
> 
> The NFS server for the mount is a vendor provided NAS device, and I'm 
> not sure what its OS or NFS server is. However, the fact that lsof shows 
> the lock has been granted seems to indicate the disconnect is somewhere 
> on the client side after getting the lock but before waking up fcntl. I 
> also run the exact some program on a Solaris host against the same mount 
> and have no issues at all, which also seems to rule out a server-side issue.
> 
> The test program I'm using is a c app compiled with gcc v3.3.1. Any 
> thoughts? 
Using your test program I see there is a window where a GRANT
message can come in before the client has added a the lock request
to the blocked locked list. In this case the status in the
GRANT reply is set to one (i.e. nlm_lck_denied), but the lock will be
retried after a 30sec time out, which can give the appearance
that the app is hung... With my testing, I've simply waited a
period of time and the app always returned....

So we can compare findings, I would like you to post a bzip2-ed
binary tethereal trace. Something like:
     tethereal -w /tmp/data.pcap host <client> and host <server>
     bzip2 /tmp/data.pcap

steved.





-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Fcntl F_SETLKW hangs on Linux NFS mount

[khowe]

>> I'm running into an issue when using fcntl with the F_SETLKW command
on=20
>> a Linux NFS client. In cases where I have many processes all trying
to=20
>> get a write lock at once, the call to fcntl never returns, even
though=20
>> the lock appears to have been granted successfully. It is not a=20
>> consistent error, but it is fairly repeatable (about 30% of the
time).=20
>> I'm running on a RedHat client, though I'm not exactly sure of the=20
>> RedHat version. However, the kernel reported by uname is "Linux=20
>> prblnx-bo04 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:17:59 EDT 2005 i686=20
>> i686 i386 GNU/Linux", and it appears to be using v1.0.6 of nfs-utils.

>> The mounts are set via autofs, using automount v4.1.3-130.
>>=20
>> My test program creates 16 process that all try to get a write lock
on=20
>> the same file, and it will frequently fail such that one of the=20
>> processes appears to have been granted the lock but does not return
from=20
>> the fcntl call (and the remaining processes are then also all stuck).

>> Listing open files and locks using "lsof -N" shows that one of the=20
>> processes has been granted the W lock, yet this same process does not

>> return from the fcntl call. I have toyed with the mounting options on

>> the client and I am seeing this behavior with NFS v2 and v3, with
short=20
>> or long timeouts (from timeo=3D7 to timeo=3D600), with intr and with
nointr,=20
>> and with r and wsize's ranging from 1024 to 32768. The only option I=20
>> haven't been able to mess with is using TCP instead of UDP, since
this=20
>> particular RedHat build does not seem to allow the TCP option.
>
> Using your test program I see there is a window where a GRANT
> message can come in before the client has added a the lock request
> to the blocked locked list. In this case the status in the
> GRANT reply is set to one (i.e. nlm_lck_denied), but the lock will be
> retried after a 30sec time out, which can give the appearance
> that the app is hung... With my testing, I've simply waited a
> period of time and the app always returned....
>=20
> So we can compare findings, I would like you to post a bzip2-ed
> binary tethereal trace. Something like:
>      tethereal -w /tmp/data.pcap host <client> and host <server>
>      bzip2 /tmp/data.pcap

I have also seen this 30 second timeout behavior while testing, in
addition to the full-fledged hang that I described. In the case of a 30
second timeout, one symptomatic difference is that when I do an lsof -N,
none of the processes show a "W" lock on the file. After 30 seconds, one
of them will apparently retry, get the lock, and all is well. In the
other case, however, an lsof -N shows that one of the processes actually
has a "W" lock on the file, yet it never returns (where "never" is at
least 6 hours or so, which is the longest I've left it in this state).
Anyhow, I will try to get tethereal traces for both of these scenarios.
Thanks...

- Keith


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs