nfs client: Now you see it, now you don't

[Larry Keegan <lk@pfw.demon.co.uk>]

Dear Chaps,

I am experiencing some inexplicable NFS behaviour which I would like to
run past you.

I have a linux NFS server running kernel 3.10.2 and some clients
running the same. The server is actually a pair of identical
machines serving up a small number of ext4 filesystems atop drbd. They
don't do much apart from serve home directories and deliver mail
into them. These have worked just fine for aeons.

The problem I am seeing is that for the past month or so, on and off,
one NFS client starts reporting stale NFS file handles on some part of
the directory tree exported by the NFS server. During the outage the
other parts of the same export remain unaffected. Then, some ten
minutes to an hour later they're back to normal. Access to the affected
sub-directories remains possible from the server (both directly and via
nfs) and from other clients. There do not appear to be any errors on
the underlying ext4 filesystems.

Each NFS client seems to get the heebie-jeebies over some directory or
other pretty much independently. The problem affects all of the
filesystems exported by the NFS server, but clearly I notice it first
in home directories, and in particular in my dot subdirectories for
things like my mail client and browser. I'd say something's up the
spout about 20% of the time.

The server and clients are using nfs4, although for a while I tried
nfs3 without any appreciable difference. I do not have CONFIG_FSCACHE
set.

I wonder if anyone could tell me if they have ever come across this
before, or what debugging settings might help me diagnose the problem?

Yours,

Larry