HELP: NFS mount hangs when attempting to copy file

HELP: NFS mount hangs when attempting to copy file

[Timothy Miller]

My research suggests that NFS client mounting is kernel-based, so
that's why I'm posting here.  If there's a more appropriate list to
post to, I apologise, but I am not a list member.

I'm having a bit of a problem doing simple copies over an NFS mount. 
The client is running Linux (2.6.11), and the server is running
Solaris (5.8).

When I first boot the client, getting NFS directory listings works
just fine.  But the instant I try to copy a file (to or from), the NFS
mount hangs.  While I can still do other network activity (even rlogin
to the server), any NFS access I try to do after that point hangs,
including directory listings.

I have had this same client and server working flawlessly for years. 
The only change is that the client is now on a VPN (Watchguard SOHO
box).  However, I have a Sun machine on the same VPN network segment,
and it can copy files with no problem, so it's not the router/SOHO
that's blocking anything.  (NIS and DNS also work just fine for both
machines.)

Also, after it hangs like that, I cannot reboot the machine normally. 
When attempting to unmount the network filesystems, the shutdown
hangs, and I have to hard-reset the machine.
 
Is there anyone who could please help me to debug this problem?  As
far as I know, I have NFS setup properly, but I don't know enough
about it to know what options I might try.  I don't even care if the
fix degrades performance; I just want it to not hang.

Does anyone have any ideas? 
 
Thanks very much in advance!

Re: HELP: NFS mount hangs when attempting to copy file

[Trond Myklebust]

on den 20.07.2005 Klokka 18:56 (-0400) skreiv Timothy Miller:
> My research suggests that NFS client mounting is kernel-based, so
> that's why I'm posting here.  If there's a more appropriate list to
> post to, I apologise, but I am not a list member.
> 
> I'm having a bit of a problem doing simple copies over an NFS mount. 
> The client is running Linux (2.6.11), and the server is running
> Solaris (5.8).
> 
> When I first boot the client, getting NFS directory listings works
> just fine.  But the instant I try to copy a file (to or from), the NFS
> mount hangs.  While I can still do other network activity (even rlogin
> to the server), any NFS access I try to do after that point hangs,
> including directory listings.
> 
> I have had this same client and server working flawlessly for years. 
> The only change is that the client is now on a VPN (Watchguard SOHO
> box).  However, I have a Sun machine on the same VPN network segment,
> and it can copy files with no problem, so it's not the router/SOHO
> that's blocking anything.  (NIS and DNS also work just fine for both
> machines.)

I beg to disagree. A lot of these VPN solutions are unfriendly to MTU
path discovery over UDP. Sun uses TCP by default when mounting NFS
partitions. Have you tried this on your Linux box?

Cheers,
  Trond

> Also, after it hangs like that, I cannot reboot the machine normally. 
> When attempting to unmount the network filesystems, the shutdown
> hangs, and I have to hard-reset the machine.
>  
> Is there anyone who could please help me to debug this problem?  As
> far as I know, I have NFS setup properly, but I don't know enough
> about it to know what options I might try.  I don't even care if the
> fix degrades performance; I just want it to not hang.
> 
> Does anyone have any ideas? 
>  
> Thanks very much in advance!
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: HELP: NFS mount hangs when attempting to copy file

[Timothy Miller]

On 7/23/05, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:

> I beg to disagree. A lot of these VPN solutions are unfriendly to MTU
> path discovery over UDP. Sun uses TCP by default when mounting NFS
> partitions. Have you tried this on your Linux box?

I changed the protocol to TCP and changed rsize and wsize to 1024.  I
don't know which of those fixed it, but I'm going to leave it for now.

As for MTU, yeah, the Watchguard box seems to have some hard-coded
limits, and for whatever reason KDE and GNOME graphical logins do
something that exceeds those limits, completely independent of NFS,
and hang up hard.

Thanks.

Re: HELP: NFS mount hangs when attempting to copy file

[Michael Clark]

Timothy Miller wrote:

>On 7/23/05, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
>  
>
>>I beg to disagree. A lot of these VPN solutions are unfriendly to MTU
>>path discovery over UDP. Sun uses TCP by default when mounting NFS
>>partitions. Have you tried this on your Linux box?
>>    
>>
>
>I changed the protocol to TCP and changed rsize and wsize to 1024.  I
>don't know which of those fixed it, but I'm going to leave it for now.
>
>As for MTU, yeah, the Watchguard box seems to have some hard-coded
>limits, and for whatever reason KDE and GNOME graphical logins do
>something that exceeds those limits, completely independent of NFS,
>and hang up hard.
>  
>

If possible it would also be good to fix the misconfigured VPN box
that's breaking the PMTU discovery if you can (usually it's too
aggressive blocking of ICMP messages). Although a wsize/rzise of 1024
and using TCP probably makes sense for NFS over a VPN (avoid
framentation overhead and let TCP handle the retransmission).

My guess is the Watchguard is blocking ICMP "fragmentation needed"
messages (which is resulting in the PMTU discovery breakage).

Enabling ICMP "fragmentation needed" messages to pass through the VPNs
firewall if you can should fix your other problems aswell.

~mc

RE: HELP: NFS mount hangs when attempting to copy file

[Roger Heflin]

A kde and gnome are well above MTU they don't know anything
about MTU and neither does NFS, if those hang it up you have
a network configuration problem, and should probably fix it, 
as a number of other things will show the problem also.

Routers almost always have hard coded MTU limits, and they are
almost never the default 1500, so everything needs to be
properly told what your networks MTU is, or some external
device needs to be taking care of it properly.

                    Roger

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of 
> Timothy Miller
> Sent: Saturday, July 23, 2005 9:52 PM
> To: Trond Myklebust
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: HELP: NFS mount hangs when attempting to copy file
> 
> On 7/23/05, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> 
> > I beg to disagree. A lot of these VPN solutions are 
> unfriendly to MTU 
> > path discovery over UDP. Sun uses TCP by default when mounting NFS 
> > partitions. Have you tried this on your Linux box?
> 
> I changed the protocol to TCP and changed rsize and wsize to 
> 1024.  I don't know which of those fixed it, but I'm going to 
> leave it for now.
> 
> As for MTU, yeah, the Watchguard box seems to have some 
> hard-coded limits, and for whatever reason KDE and GNOME 
> graphical logins do something that exceeds those limits, 
> completely independent of NFS, and hang up hard.
> 
> Thanks.
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>