Re: Stale File handles keep coming back

[Frank Steiner <fsteiner-mail@bio.ifi.lmu.de>]

[-- Attachment #1: Type: text/plain, Size: 6254 bytes --]

Hi,

while I was still trying to set up some test environments,
one of our users stepped again on two stale directories, so
I fetched all the info you mentioned. The stale NFS occured
in the users home (mounted from /export/home from the server).


Neil Brown wrote

 >>>  echo 2048 > /proc/sys/sunrpc/rpc_debug
 >>>  grep . /proc/net/rpc/*/content

This is a lot because we have 60 NFS clients. I just grepped the
lines for the client on which the stale NFS occured at that moment
which is still a lot due to the many mounts:

/proc/net/rpc/auth.unix.ip/content-# expiry=2147483647 refcnt=0
/proc/net/rpc/auth.unix.ip/content:nfsd 141.84.1.156 cauchy.bio.ifi.lmu.de
--
/proc/net/rpc/nfsd.export/content-# expiry=2147483647 refcnt=1
/proc/net/rpc/nfsd.export/content:/ 
cauchy.bio.ifi.lmu.de(ro,no_root_squash,sync,wdelay)
--
/proc/net/rpc/nfsd.export/content-# expiry=2147483647 refcnt=1
/proc/net/rpc/nfsd.export/content:/export/clientpass 
cauchy.bio.ifi.lmu.de(ro,no_root_squash,sync,wdelay)
--
/proc/net/rpc/nfsd.export/content-# expiry=2147483647 refcnt=1
/proc/net/rpc/nfsd.export/content:/export/clientroot 
cauchy.bio.ifi.lmu.de(rw,no_root_squash,sync,wdelay)
--
/proc/net/rpc/nfsd.export/content-# expiry=2147483647 refcnt=1
/proc/net/rpc/nfsd.export/content:/var  cauchy.bio.ifi.lmu.de(ro,root_squash,sync,wdelay)
--
/proc/net/rpc/nfsd.export/content-# expiry=2147483647 refcnt=1
/proc/net/rpc/nfsd.export/content:/export/diskless/141.84.1.156 
cauchy.bio.ifi.lmu.de(rw,no_root_squash,sync,wdelay)
--
/proc/net/rpc/nfsd.export/content-# expiry=2147483647 refcnt=1
/proc/net/rpc/nfsd.export/content:/export/home 
cauchy.bio.ifi.lmu.de(rw,root_squash,sync,wdelay)
--
/proc/net/rpc/nfsd.fh/content-# expiry=2147483647 refcnt=0
/proc/net/rpc/nfsd.fh/content:cauchy.bio.ifi.lmu.de 0 0x0200080000000002 /var
--
/proc/net/rpc/nfsd.fh/content-# expiry=2147483647 refcnt=0
/proc/net/rpc/nfsd.fh/content:cauchy.bio.ifi.lmu.de 0 0x0100fc0000000004 
/export/clientpass
--
/proc/net/rpc/nfsd.fh/content-# expiry=2147483647 refcnt=0
/proc/net/rpc/nfsd.fh/content:cauchy.bio.ifi.lmu.de 0 0x0100080000000002 /
--
/proc/net/rpc/nfsd.fh/content-# expiry=2147483647 refcnt=0
/proc/net/rpc/nfsd.fh/content:cauchy.bio.ifi.lmu.de 0 0x0100fc000000000b 
/export/clientroot
--
/proc/net/rpc/nfsd.fh/content-# expiry=2147483647 refcnt=0
/proc/net/rpc/nfsd.fh/content:cauchy.bio.ifi.lmu.de 0 0x0100fc0000002e4a 
/export/diskless/141.84.1.156
--
/proc/net/rpc/nfsd.fh/content-# expiry=2147483647 refcnt=0
/proc/net/rpc/nfsd.fh/content:cauchy.bio.ifi.lmu.de 0 0x0000fc0000000002 /export/home


 >>>  ls -l /proc/fs/nfsd

babbage /root/tmp# ls -l /proc/fs/nfsd
total 0
dr-xr-xr-x    2 root     root            0 Apr 29 09:44 .
dr-xr-xr-x    4 root     root            0 Apr 29 09:44 ..

? Should there be sth?


 >>>  cat /proc/fs/nfs/exports

babbage /root/tmp# cat /proc/fs/nfs/exports |grep cauchy
/       cauchy.bio.ifi.lmu.de(ro,no_root_squash,sync,wdelay)
/export/clientpass      cauchy.bio.ifi.lmu.de(ro,no_root_squash,sync,wdelay)
/export/clientroot      cauchy.bio.ifi.lmu.de(rw,no_root_squash,sync,wdelay)
/var    cauchy.bio.ifi.lmu.de(ro,root_squash,sync,wdelay)
/export/diskless/141.84.1.156   cauchy.bio.ifi.lmu.de(rw,no_root_squash,sync,wdelay)
/export/home    cauchy.bio.ifi.lmu.de(rw,root_squash,sync,wdelay)


Additionally, here are the relevant mount options on the client:

cauchy /root# mount | grep 141.84.1.131
/dev/root on / type nfs 
(rw,v3,rsize=16384,wsize=16384,hard,intr,tcp,nolock,addr=141.84.1.131)
141.84.1.131://export/diskless/141.84.1.156//etc/local on /etc/local type nfs 
(rw,v3,rsize=16384,wsize=16384,hard,intr,tcp,nolock,addr=141.84.1.131)
141.84.1.131://export/diskless/141.84.1.156//var on /var type nfs 
(rw,v3,rsize=16384,wsize=16384,hard,intr,tcp,nolock,addr=141.84.1.131)
141.84.1.131:/boot on /boot type nfs 
(ro,tcp,hard,intr,rsize=16384,wsize=16384,addr=141.84.1.131)
141.84.1.131:/var/adm on /var/adm type nfs 
(ro,tcp,hard,intr,rsize=16384,wsize=16384,addr=141.84.1.131)
141.84.1.131:/var/lib/texmf on /var/lib/texmf type nfs 
(ro,tcp,hard,intr,rsize=16384,wsize=16384,addr=141.84.1.131)
141.84.1.131:/var/lib/rpm on /var/lib/rpm type nfs 
(ro,tcp,hard,intr,rsize=16384,wsize=16384,addr=141.84.1.131)
141.84.1.131:/var/log/apache2 on /var/httpd type nfs 
(ro,tcp,hard,intr,rsize=16384,wsize=16384,addr=141.84.1.131)
141.84.1.131:/export/clientpass on /export type nfs 
(ro,tcp,hard,intr,rsize=16384,wsize=16384,addr=141.84.1.131)
141.84.1.131:/export/clientroot on /export/localhome/root type nfs 
(rw,tcp,hard,intr,rsize=16384,wsize=16384,addr=141.84.1.131)
babbage:/export/home on /home type nfs 
(rw,tcp,hard,rsize=16384,wsize=16384,addr=141.84.1.131)



 > So, when it happens again, please check that the IP address of the
 > client really is in /proc/net/rpc/auth.unix.ip/content and then

Yes, it was definitely there:
babbage /root/tmp# grep 141.84.1.156 /proc/net/rpc/auth.unix.ip/content
nfsd 141.84.1.156 cauchy.bio.ifi.lmu.de


 >
 >   echo 1023 > /proc/sys/sunrpc/nfsd_debug
 > on the server.
 > Then on the client,
 >   cd $HOME
 >   ls -l the/offending/directory

This is amazing: This ls does work, while the one in the shell
where the stale occured still don't work. So the directory
is only stale in on shell and not stale in the other shell.
I thought that the stale should would have recovered, too...

 > See if that works, and see what you get in the kernel logs.

We did the following:
1) echo 1023 > /proc/sys/sunrpc/nfsd_debug
2) the user did a ls in the stale directory, and got the stale message
    again
3) the user did a "ls <the directory>" from another shell and got
    the contents of the directory
4) the user did a ls in the stale shell and go the stale messages
    again
5) echo 0 > /proc/sys/sunrpc/nfsd_debug

So I hope that the log does not contain too much other, useless
information.


Let me know if I can provide more information!

cu,
Frank

-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

[-- Attachment #2: nfsd.debug.bz2 --]
[-- Type: application/x-bzip, Size: 4106 bytes --]