2.6.32, NFSv4, "Stale NFS file handle" problem

2.6.32, NFSv4, "Stale NFS file handle" problem

[Nick Patavalis]

Hi,

I don't know if this is an NFS bug, but seems a lot like it... My
server-host exports a few NTFS filesystems (read-only exports) using
NFSv4. Two client-hosts mount these file-systems.

Client A runs 2.6.31, and client B 2.6.32. Client A works fine. Client
B, while it initially mounts everything ok, and for some time
everything works ok (or so it seems), after a while it starts
returning "Stale NFS file hande" errors for random files and
directories. Client A NEVER returns stale errors, EVEN after a server
reboot.

Has anyone seen something like this before? Is it a known problem?
Who's to blame (probably me... but anyway).

Any additional information you may require I will try to provide.

Thanks in advance.
/npat

Some additional details:

On the server

  # uname -a

  Linux moviesserver 2.6.32-26-generic #48-Ubuntu SMP Wed Nov 24
  09:00:03 UTC 2010 i686 GNU/Linux

  # cat /etc/exports

  /mnt             *(fsid=0,ro,crossmnt,async,insecure,
                               all_squash,no_subtree_check)
  /mnt/mydvds8     *(fsid=1,ro,async,insecure,all_squash,no_subtree_check)
  /mnt/mydvds1-2-7 *(fsid=2,ro,async,insecure,all_squash,no_subtree_check)
  /mnt/mydvds3-4   *(fsid=3,ro,async,insecure,all_squash,no_subtree_check)
  /mnt/mydvds5-6   *(fsid=4,ro,async,insecure,all_squash,no_subtree_check)
  /mnt/mydvds9     *(fsid=5,ro,async,insecure,all_squash,no_subtree_check)

The /mnt/mydvdsXXX directories are mountpoints of NTFS filesystems on
USB drives (if this has any relevance).

On client A:

  # uname -a

  Linux XBMCLive 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31
  UTC 2009 i686 GNU/Linux

  # cat /etc/fstab

  192.168.150.2:/mydvds1-2-7 /mnt/thepat/mydvds1-2-7 nfs4 soft,ro,
      nocto,actimeo=300 0 0
  192.168.150.2:/mydvds3-4 /mnt/thepat/mydvds3-4 nfs4 soft,ro,
      nocto,actimeo=300 0 0
  192.168.150.2:/mydvds5-6 /mnt/thepat/mydvds5-6 nfs4
soft,ro,nocto,actimeo=300 0 0
  192.168.150.2:/mydvds8 /mnt/thepat/mydvds8 nfs4 soft,ro,
      nocto,actimeo=300 0 0
  192.168.150.2:/mydvds9 /mnt/thepat/mydvds9 nfs4 soft,ro,
      nocto,actimeo=300 0 0

On Client B:

  # uname -a

  Linux azure 2.6.32-28-generic #55-Ubuntu SMP Mon Jan 10 21:21:01 UTC
  2011 i686 GNU/Linux

  # cat /etc/fstab

  ... same as above ...

On client B, after the problem appears:

  # ls /mnt/thepat/*
  /mnt/thepat/mydvds1-2-7:
  msdownld.tmp  MyDVDs1  MyDVDs2  MyDVDs7  RECYCLER  System Volume Information

  /mnt/thepat/mydvds3-4:
  MyDVDs3  MyDVDs4  RECYCLER  System Volume Information
  ls: cannot access /mnt/thepat/mydvds5-6/MyDVDs5: Stale NFS file handle
  ls: cannot access /mnt/thepat/mydvds5-6/MyDVDs6: Stale NFS file handle
  ls: cannot access /mnt/thepat/mydvds5-6/Recycled: Stale NFS file handle
  ls: cannot access /mnt/thepat/mydvds5-6/RECYCLER: Stale NFS file handle
  ls: cannot access /mnt/thepat/mydvds5-6/System Volume Information:
Stale NFS file handle

  ... etc ...

Re: 2.6.32, NFSv4, "Stale NFS file handle" problem

[Nick Patavalis]

Hi,

On Mon, Feb 21, 2011 at 2:54 AM, Nick Patavalis <npat@efault.net> wrote:
>
> I don't know if this is an NFS bug, but seems a lot like it... My
> server-host exports a few NTFS filesystems (read-only exports) using
> NFSv4. Two client-hosts mount these file-systems.
>
> Client A runs 2.6.31, and client B 2.6.32. Client A works fine. Client
> B, while it initially mounts everything ok, and for some time
> everything works ok (or so it seems), after a while it starts
> returning "Stale NFS file hande" errors for random files and
> directories. Client A NEVER returns stale errors, EVEN after a server
> reboot.
>

Some additional information.

After the problem appears (i.e. after "client B" starts returning
"stale file handle" errors), an "ls" on a file (directory) for
which the file-handle is stale, with "sunrpc.nfs_debug = 1023"
shows:

  # ls /mnt/thepat/mydvds5-6/MyDVDs5
  ls: cannot access /mnt/thepat/mydvds5-6/MyDVDs5: Stale NFS file
  handle
  # ls /mnt/thepat/mydvds5-6/MyDVDs5
  ls: cannot access /mnt/thepat/mydvds5-6/MyDVDs5: Stale NFS file
  handle

While in syslog:

  Feb 21 10:59:19 azure kernel: [197436.952297] encode_compound: tag=
  Feb 21 10:59:19 azure kernel: [197436.957441] decode_attr_type: type=040000
  Feb 21 10:59:19 azure kernel: [197436.957448] decode_attr_change:
change attribute=381218604654685448
  Feb 21 10:59:19 azure kernel: [197436.957454] decode_attr_size: file size=4096
  Feb 21 10:59:19 azure kernel: [197436.957460] decode_attr_fsid: fsid=(0x4/0x0)
  Feb 21 10:59:19 azure kernel: [197436.957465] decode_attr_fileid: fileid=5
  Feb 21 10:59:19 azure kernel: [197436.957471]
decode_attr_fs_locations: fs_locations done, error = 0
  Feb 21 10:59:19 azure kernel: [197436.957477] decode_attr_mode: file mode=0777
  Feb 21 10:59:19 azure kernel: [197436.957481] decode_attr_nlink: nlink=1
  Feb 21 10:59:19 azure kernel: [197436.957489] decode_attr_owner:
nfs_map_name_to_uid failed!
  Feb 21 10:59:19 azure kernel: [197436.957494] decode_attr_owner: uid=-2
  Feb 21 10:59:19 azure kernel: [197436.957501] decode_attr_group:
nfs_map_group_to_gid failed!
  Feb 21 10:59:19 azure kernel: [197436.957506] decode_attr_group: gid=-2
  Feb 21 10:59:19 azure kernel: [197436.957511] decode_attr_rdev: rdev=(0x0:0x0)
  Feb 21 10:59:19 azure kernel: [197436.957516]
decode_attr_space_used: space used=4096
  Feb 21 10:59:19 azure kernel: [197436.957522]
decode_attr_time_access: atime=1297893893
  Feb 21 10:59:19 azure kernel: [197436.957527]
decode_attr_time_metadata: ctime=1297893893
  Feb 21 10:59:19 azure kernel: [197436.957533]
decode_attr_time_modify: mtime=1297893893
  Feb 21 10:59:19 azure kernel: [197436.957538]
decode_attr_mounted_on_fileid: fileid=0
  Feb 21 10:59:19 azure kernel: [197436.957543] decode_getfattr: xdr returned 0
  Feb 21 10:59:19 azure kernel: [197436.957557] NFS:
nfs_update_inode(0:1e/5 ct=3 info=0x27e67)
  Feb 21 10:59:19 azure kernel: [197436.957567] NFS:
permission(0:1e/5), mask=0x1, res=0
  Feb 21 10:59:19 azure kernel: [197436.957578] NFS:
nfs_lookup_revalidate(/MyDVDs5) is valid
  Feb 21 10:59:19 azure kernel: [197436.957586] NFS: revalidating (0:1e/33)
  Feb 21 10:59:19 azure kernel: [197436.957596] encode_compound: tag=
  Feb 21 10:59:19 azure kernel: [197436.963413] nfs_revalidate_inode:
(0:1e/33) getattr failed, error=-116
  Feb 21 10:59:19 azure kernel: [197436.963422] NFS: dentry_delete(/MyDVDs5, 88)
  Feb 21 10:59:28 azure kernel: [197446.008528] NFS:
permission(0:1e/5), mask=0x1, res=0
  Feb 21 10:59:28 azure kernel: [197446.008540] NFS:
nfs_lookup_revalidate(/MyDVDs5) is valid
  Feb 21 10:59:28 azure kernel: [197446.008547] NFS: revalidating (0:1e/33)
  Feb 21 10:59:28 azure kernel: [197446.008568] encode_compound: tag=
  Feb 21 10:59:28 azure kernel: [197446.015965] nfs_revalidate_inode:
(0:1e/33) getattr failed, error=-116
  Feb 21 10:59:28 azure kernel: [197446.015974] NFS: dentry_delete(/MyDVDs5, 88)

Re: 2.6.32, NFSv4, "Stale NFS file handle" problem

[Nick Patavalis]

Hi,

On Mon, Feb 21, 2011 at 11:09 AM, Nick Patavalis <npat@efault.net> wrote:
>
> On Mon, Feb 21, 2011 at 2:54 AM, Nick Patavalis <npat@efault.net> wrote:
>>
>> I don't know if this is an NFS bug, but seems a lot like it... My
>> server-host exports a few NTFS filesystems (read-only exports) using
>> NFSv4. Two client-hosts mount these file-systems.
>>
>> Client A runs 2.6.31, and client B 2.6.32. Client A works fine. Client
>> B, while it initially mounts everything ok, and for some time
>> everything works ok (or so it seems), after a while it starts
>> returning "Stale NFS file hande" errors for random files and
>> directories. Client A NEVER returns stale errors, EVEN after a server
>> reboot.
>>
>
> Some additional information.
>
> After the problem appears (i.e. after "client B" starts returning
> "stale file handle" errors), an "ls" on a file (directory) for
> which the file-handle is stale, with "sunrpc.nfs_debug = 1023"
> shows:
>

I also tried:

- Exporting /mnt (the NFSv4 preudo-root) without the "crossmount"
  option (and having the clients mount the subordinate file-systems)

- Exporting /mnt with "crossmount" and having the client B (the
  "problematic" one, running kernel 2.6.32) mount "server:/" instead
  of the subordinate filesystems.

- Making client B mount with options: "ro,soft,proto=tcp"
  (i.e. without the "nocto,actimeo=300")

No observable change in behavior. Client A (the one with kernel
2.6.31) works perfectly. Client B (the one with kernel 2.6.32) mounts
ok but, at some point in time, it starts returning "stale file handle"
errors (as described in the previous message).

At this point I start to run out of things to try! Any suggestions
would be greately appreciated.

Thanks again
/npat