nfs - duplicate directory entries

nfs - duplicate directory entries

[Vladimir Volovich]

Hi!

we have an NFS server (windows-based), and mount a directory from this
server on a linux box; there is some bug appearing sometimes, namely
some of the directory entries are reported twice:

$ ls | wc
   5968    5968   41776
$ ls | uniq | wc
   5964    5964   41748
$ ls | uniq -d
112920
112921
112922
112923
$ ls -li | grep '11292[0123]'
8507232 drwx------ 2 web web 64 2007-02-09 10:54 112920
8507232 drwx------ 2 web web 64 2007-02-09 10:54 112920
8507219 drwx------ 2 web web 64 2007-02-09 10:54 112921
8507219 drwx------ 2 web web 64 2007-02-09 10:54 112921
8507195 drwx------ 2 web web 64 2007-02-09 10:54 112922
8507195 drwx------ 2 web web 64 2007-02-09 10:54 112922
8507188 drwx------ 2 web web 64 2007-02-09 10:54 112923
8507188 drwx------ 2 web web 64 2007-02-09 10:54 112923

i.e. the current directory contains sub-directory 112920 twice,
sub-directory 112921 twice, etc. - and the inode numbers are the same.

i wonder if you have ever seen such strange behavior?  sometimes this
behavior is cured by unmounting and remounting the directory; but it
occurs regularly and reliably.

this does not happen if the directory from the same NFS server is
mounted on FreeBSD box.

linux version is 2.6.22 (from debian "sid").

the following information may of may not be related to the above bug:
the nfs server always reports link count for any directory as 2,
regardless of the number of subdirectories. (so e.g. find(1) says one
should use the -noleaf option).

if you need me to provide more information, please let me know.

Best,
v.

Re: nfs - duplicate directory entries

[Trond Myklebust]

On Fri, 2007-08-10 at 23:21 +0400, Vladimir Volovich wrote:
> Hi!
> 
> we have an NFS server (windows-based), and mount a directory from this
> server on a linux box; there is some bug appearing sometimes, namely
> some of the directory entries are reported twice:
> 
> $ ls | wc
>    5968    5968   41776
> $ ls | uniq | wc
>    5964    5964   41748
> $ ls | uniq -d
> 112920
> 112921
> 112922
> 112923
> $ ls -li | grep '11292[0123]'
> 8507232 drwx------ 2 web web 64 2007-02-09 10:54 112920
> 8507232 drwx------ 2 web web 64 2007-02-09 10:54 112920
> 8507219 drwx------ 2 web web 64 2007-02-09 10:54 112921
> 8507219 drwx------ 2 web web 64 2007-02-09 10:54 112921
> 8507195 drwx------ 2 web web 64 2007-02-09 10:54 112922
> 8507195 drwx------ 2 web web 64 2007-02-09 10:54 112922
> 8507188 drwx------ 2 web web 64 2007-02-09 10:54 112923
> 8507188 drwx------ 2 web web 64 2007-02-09 10:54 112923
> 
> i.e. the current directory contains sub-directory 112920 twice,
> sub-directory 112921 twice, etc. - and the inode numbers are the same.

That can happen if the NFS server doesn't send unique cookies.

Trond

Re: nfs - duplicate directory entries

[Vladimir Volovich]

"TM" == Trond Myklebust writes:

 >> i.e. the current directory contains sub-directory 112920 twice,
 >> sub-directory 112921 twice, etc. - and the inode numbers are the
 >> same.

 TM> That can happen if the NFS server doesn't send unique cookies.

i have 2 questions:

1) how to debug this, to confirm that this is caused by NFS server not
   sending unique cookies? (to try to report to the "vendor" of the
   NFS server)

2) does it make sence to try to gracefully work-around this problem on
   the linux side? or would it significantly complicate things?

Best,
v.

Re: nfs - duplicate directory entries

[Trond Myklebust]

On Sat, 2007-08-11 at 01:05 +0400, Vladimir Volovich wrote:
> "TM" == Trond Myklebust writes:
> 
>  >> i.e. the current directory contains sub-directory 112920 twice,
>  >> sub-directory 112921 twice, etc. - and the inode numbers are the
>  >> same.
> 
>  TM> That can happen if the NFS server doesn't send unique cookies.
> 
> i have 2 questions:
> 
> 1) how to debug this, to confirm that this is caused by NFS server not
>    sending unique cookies? (to try to report to the "vendor" of the
>    NFS server)

You would have to use something like 'wireshark' or 'ethereal' to look
at the actual cookies returned by the server in reply to client READDIR
requests.

> 2) does it make sence to try to gracefully work-around this problem on
>    the linux side? or would it significantly complicate things?

Assuming that it is indeed the non-unique cookie problem:

Firstly, it is my policy never to fix NFS server bugs inside the NFS
client.

Secondly, it is in any case impossible to work around this sort of thing
reliably. The cookie is used by the client in the case where you have a
large directory that requires several READDIR rpc calls. It acts in much
the same way telldir()/seekdir() does: the server supplies the cookie in
a previous READDIR call, then upon the next call to READDIR, the client
uses the cookie to tells the server to first seekdir() to the location
that was last read, and then to resume from there.

Trond