Hard to debug NFS loss of connectivity

[Emmanuel Florac <eflorac@intellique.com>]

Hi list, I have a serious problem I've never met before. Here is the
setup:

The NFS server is running Debian 6 amd64 but with a plain vanilla 3.2.50
kernel. it shares a large 81 TB volume (XFS over LVM on hardware RAID6)
through nfs without any particular options. Here is a glimpse
of /etc/exports:

/mnt/raid 10.1.1.0/255.255.255.128(fsid=1,rw,no_root_squash,async,no_subtree_check)

On the other side is a VMWare ESX VM running Ubuntu 12.04LTS, kernel 3.2.0-52 Ubuntu 
amd64 mounting the share. From the fstab:

10.1.1.99:/mnt/raid          /server         nfs  rw,hard,intr            0    0

The problem is as follow: stat'ing files on the VM makes the 
NFS connection drop. For instance:

find /server -type f -ls

It works for a while, then stops responding. The NFS mount is frozen. 
The network link is OK; I still can ssh from the server to the VM 
and back, I can wget from the VM to the server, ping the server 
from the VM, etc. Only NFS is affected.

Restarting NFS on the server does nothing to unfreeze the mount. 
Using nfs4 instead of nfs3 does nothing. The only remedy is to reboot the VM.
There isn't any error in dmesg, /var/log/syslog or 
/var/log/messages in the VM nor the server.

I've tried rebooting the server on a 3.9.7 kernel. Same thing. 
Of course there isn't any data corruption of any sort. 
Running "find /mnt/raid -type f -ls" on the server works 
perfectly and lists about 25000 files without the slightest trouble.

It works equally well if I mount the NFS share on the server itself.


Now it's becoming crazier: When I run the find command as
previously said, it freezes always on the same file, for
instance :

/server/folder1/folder2/folder3/folder4/.svn/somefile

However, if after a fresh reboot I do

stat /server/folder1/folder2/folder3/folder4/.svn/somefile

no problem. Even doing this:

cd /server/folder1/folder2/folder3/folder4/ && find . -type f -ls

works. However this

cd /server/folder1/folder2/folder3/ && find . -type f -ls

doesn't fly. It freezes at exactly the same point.
In the first test (running directly from /server) it
freezes after successfully listing 10000 files. In the last
test it freezes after only 25 files. 
So apparently it's not about the number of files.


Now I'm stuck. Out of going through tcpdump, I have absolutely 
not the faintest idea about what's going on, except I tend to 
think that's some Ubuntu kernel bug.

Any hint, idea, etc would be extremely welcome. Even some
debugging method less painful than digging through huge 
tcpdumps would be nice :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------