NFS Hangs on heavy I/O

NFS Hangs on heavy I/O

[Rajat]

Hi list,

I am facing a NFS hang problem when I perform a very heavy I/O on a localhost 
mount.
The situation is:
$>mount -t nfs localhost:/home/RHL /mnt/nfs

where /home/RHL is a Ext3 Partition (/dev/sda5). Mount paramter was 'hard' by 
default

When I perform a heavy I/O on nfs mount
$> createfile /mnt/nfs/huge_file 512 1048576

Where createfile would try to write 512 blocks of 1048576 bytes each (each 
system call of 1048576, 512 times = 512MB file).

While the file is being created, I tried to ls in the /mnt/nfs directory - and 
it hanged (after creating about 75MB file). Logging from other terminals was 
not working as well.
In my dmesg I found message saying:

"nfs: server localhost not responding, still trying"

[no other NFS related message here.]

After rebooting, I tried the same thing but ls'd /home/RHL rather than nfs 
mount - it hanged again (this time it could create 332MB file). Same message 
was relayed in dmesg. This time I was able to login (though slow) and found 
that 4 nfsd thread were there in ps.
/proc/../status of all of them was "Sleeping" (for 2) or "Disk Sleep" (for 
other 2). All were averaging about 97% sleep (got from /proc/.../status)

I can reproduce this by the same procedure. 

I am running a Intel Machine (Intel(R) Pentium(R) 4 CPU 3.20GHz HT), with 1GB 
RAM, Linux-2.6.19 stock kernel, SATA driver (/dev/sda5) (default)

I tried the same thing multiple times getting the same result (machine did not 
stall always (just a couple of times)- nfs partition and the terminal on which 
I was doing work always stalled).

Can anyone help in giving pointer as to how can I find what is wrong. In case 
you need some specific info, I would be glad to give.

Rajat


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS Hangs on heavy I/O

[Trond Myklebust]

On Thu, 2007-06-28 at 15:01 +0000, Rajat wrote:
> Hi list,
> 
> I am facing a NFS hang problem when I perform a very heavy I/O on a localhost 
> mount.
> The situation is:
> $>mount -t nfs localhost:/home/RHL /mnt/nfs
> 
> where /home/RHL is a Ext3 Partition (/dev/sda5). Mount paramter was 'hard' by 
> default
> 
> When I perform a heavy I/O on nfs mount
> $> createfile /mnt/nfs/huge_file 512 1048576
> 
> Where createfile would try to write 512 blocks of 1048576 bytes each (each 
> system call of 1048576, 512 times = 512MB file).
> 
> While the file is being created, I tried to ls in the /mnt/nfs directory - and 
> it hanged (after creating about 75MB file). Logging from other terminals was 
> not working as well.
> In my dmesg I found message saying:
> 
> "nfs: server localhost not responding, still trying"

Please use a TCP mount, with a ditto appropriate value for timeo
(preferably just the default of timeo=600).

Cheers
  Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS Hangs on heavy I/O

[Peter Staubach]

Rajat wrote:
> Hi list,
>
> I am facing a NFS hang problem when I perform a very heavy I/O on a localhost 
> mount.
> The situation is:
> $>mount -t nfs localhost:/home/RHL /mnt/nfs
>
> where /home/RHL is a Ext3 Partition (/dev/sda5). Mount paramter was 'hard' by 
> default
>
> When I perform a heavy I/O on nfs mount
> $> createfile /mnt/nfs/huge_file 512 1048576
>
> Where createfile would try to write 512 blocks of 1048576 bytes each (each 
> system call of 1048576, 512 times = 512MB file).
>
> While the file is being created, I tried to ls in the /mnt/nfs directory - and 
> it hanged (after creating about 75MB file). Logging from other terminals was 
> not working as well.
> In my dmesg I found message saying:
>
> "nfs: server localhost not responding, still trying"
>
> [no other NFS related message here.]
>
> After rebooting, I tried the same thing but ls'd /home/RHL rather than nfs 
> mount - it hanged again (this time it could create 332MB file). Same message 
> was relayed in dmesg. This time I was able to login (though slow) and found 
> that 4 nfsd thread were there in ps.
> /proc/../status of all of them was "Sleeping" (for 2) or "Disk Sleep" (for 
> other 2). All were averaging about 97% sleep (got from /proc/.../status)
>
> I can reproduce this by the same procedure. 
>
> I am running a Intel Machine (Intel(R) Pentium(R) 4 CPU 3.20GHz HT), with 1GB 
> RAM, Linux-2.6.19 stock kernel, SATA driver (/dev/sda5) (default)
>
> I tried the same thing multiple times getting the same result (machine did not 
> stall always (just a couple of times)- nfs partition and the terminal on which 
> I was doing work always stalled).
>
> Can anyone help in giving pointer as to how can I find what is wrong. In case 
> you need some specific info, I would be glad to give.

This sounds like an old problem with mounting loopback.  The standard
recommendation is "don't do that, it will hang".

Peter Zijlstra has some changes for newer kernels which address this
situation.

In the meantime, either separate the client and the server or use
bind mounts.

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs