v3/tcp mounts occasionally hang for 10+ minutes

v3/tcp mounts occasionally hang for 10+ minutes

[Andrew Ryan]

Linux client running 2.4.17+NFS_ALL patch set mounting a netapp server (via 
switched 100FDx ethernet) with the following mount opts:
rw,tcp,nfsvers=3,rsize=32768,wsize=32768,intr,hard

Works great, but occasionally (every few days) the mount hangs, for about 
10 minutes each time. I am not exactly sure what causes the hung mounts but 
at times it has seemed to correlate with the machine running very low on 
virtual memory and the kernel interceding to kill processes. Other times 
this has not seemed to be the case. Either way, it doesn't seem like 
something which should be happening.

Eventually, after ~10 minutes, the mount does come back, but in the 
meantime this causes some havoc. Several of the following messages are 
written to console:
nfs_statfs: statfs error = 512

Interestingly enough, during the time when the hang occurs, other NFS 
mounts on this client from the same server work just fine. And 
rpcinfo/showmount/etc against the server from the client all work just fine 
during the time when the mount is hung.

My questions:
1) Anyone know what the hung mount behavior could be triggered and/or how 
to prevent it?
2) If the hung mount behavior couldn't be prevented, could it at least be 
mitigated somehow by lowering some timeout to a time less than 10 minutes? 
Clients can forgive 30 seconds. But a hang of 10 minutes is sure to 
generate help desk calls.


andrew


_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: v3/tcp mounts occasionally hang for 10+ minutes

[Trond Myklebust]

>>>>> " " == Andrew Ryan <andrewr@nam-shub.com> writes:

     > Linux client running 2.4.17+NFS_ALL patch set mounting a netapp
     > server (via switched 100FDx ethernet) with the following mount
     > opts:

     > rw,tcp,nfsvers=3,rsize=32768,wsize=32768,intr,hard

     > Works great, but occasionally (every few days) the mount hangs,
     > for about 10 minutes each time. I am not exactly sure what
     > causes the hung mounts but at times it has seemed to correlate
     > with the machine running very low on virtual memory and the
     > kernel interceding to kill processes. Other times this has not
     > seemed to be the case. Either way, it doesn't seem like
     > something which should be happening.

If the OOM killer is signalling the rpciod process, then that could
indeed cause the sort of problem you are describing.
If so, please could you look into the circumstances that are causing
the OOM killer to do this. Normally rpciod should never be much of a
memory hog.

Cheers,
  Trond

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: v3/tcp mounts occasionally hang for 10+ minutes

[Andrew Ryan]

At 10:36 AM 4/2/02 +0200, Trond Myklebust wrote:

>If the OOM killer is signalling the rpciod process, then that could
>indeed cause the sort of problem you are describing.
>If so, please could you look into the circumstances that are causing
>the OOM killer to do this. Normally rpciod should never be much of a
>memory hog.


No, rpciod has never been killed by the OOM killer, afaict, the other 
processes OOM killer has killed have all been logged and rpciod has never 
been in that list. rpciod is always using a small amount of RAM.

Also, I can confirm that this problem has happened when the machine was 
fairly low on memory but not low enough to invoke the OOM killer. And as I 
noted before, access to other mounts - mounted from the same NFS server - 
on the client system is not interrupted when one of these hangs occur.

I realize part of the problem is I haven't been able to reliably identify 
exactly what is  causing the problem, but I was hoping someone with more 
knowledge of the client code or someone who had seen this problem before 
could point me in the right direction.


Thanks,
andrew


_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs