NFS hang in 2.6.21.1

NFS hang in 2.6.21.1

[John Frisk]

Team,
I have been attempting to put together a linux HA NFS
server for my home environment.  I am currently
running two Debian etch 4.0 machines both running
vanilla 2.6.21.1 kernels that I compiled (exception
being the nvidia driver for video).  When I attempt to
do some performance benchmarks, a test of bonnie++
makes the NFS client stuck in a non-responsive
non-killable state.  The NFS server at this point is
not busy doing anything so I believe there may be some
condition leading to a deadlock in the client. 

jfrisk@evil:~$ df -k .
Filesystem           1K-blocks      Used Available
Use% Mounted on
imezru-ha1:/hafs/nethome/jfrisk
                      76791424    409856  75601408  
1% /nethome/jfrisk
jfrisk@evil:~$ mount

imezru-ha1:/hafs/nethome/jfrisk on /nethome/jfrisk
type nfs (rw,hard,intr,bg,addr=192.168.69.7)

jfrisk@evil:~$ /usr/sbin/bonnie++ -f -s 100 -n 1 -r 0
Writing intelligently...done
Rewriting...                   

This is where the bonnie program hangs.
  frisk@evil:~$ ps aux | grep bonnie
jfrisk    6882  0.7  0.0   2708  1012 pts/3    D+  
09:11   0:00 /usr/sbin/bonnie++ -f -s 100 -n 1 -r 0

Top shows the user load at 1.00 (before starting is
0.10) but nothing else  of importance is running on
the system. 

I can reproduce this bug pretty consistently. I have
these machines for use by anyone on the team so I
would appreciate being guided on what to do next to
help debug this issue.  Thanks in advance!            
     


 
____________________________________________________________________________________
Now that's room service!  Choose from over 150,000 hotels
in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS hang in 2.6.21.1

[Trond Myklebust]

On Mon, 2007-05-21 at 07:18 -0700, John Frisk wrote:
> Team,
> I have been attempting to put together a linux HA NFS
> server for my home environment.  I am currently
> running two Debian etch 4.0 machines both running
> vanilla 2.6.21.1 kernels that I compiled (exception
> being the nvidia driver for video).  When I attempt to
> do some performance benchmarks, a test of bonnie++
> makes the NFS client stuck in a non-responsive
> non-killable state.  The NFS server at this point is
> not busy doing anything so I believe there may be some
> condition leading to a deadlock in the client. 

Could you please try the patches

http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-002-no_congestion_wait_in_update_request.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-003-cleanup_coalesce.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-004-cleanup_coalesce2.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-005-cleanup_readpages.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-006-fix_dirtying_race.dif
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-007-fix_page_overflow.dif

and

http://client.linux-nfs.org/Linux-2.6.x/2.6.22-rc2/linux-2.6.22-001-write_deadlock_avoidance.dif

That fixes all the write deadlock issues for me.

Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS hang in 2.6.21.1

[John Frisk]

Hi Trond,
I applied your patches to both kernels, rebooted and
still the same problem.  Once bonnie attempts to
"rewrite" it's sample file, the client machine process
cannot be killed.  The client OS however can still do
other things, such as load firefox, etc.  The
interesting thing being, it can also in another shell
access the same directory under NFS as the bonnie
program was writing in the other shell.  Sorry for not
knowing all the internals, but is the NFS client
multi-threaded too?

Output:
jfrisk@evil:~$ /usr/sbin/bonnie++ -f -s 100 -n 1 -r 0
Writing intelligently...done
Rewriting...

jfrisk@evil:~$ ps axu | grep bonnie
jfrisk    5612  0.2  0.0   2708  1012 pts/2    D+  
13:24   0:01 /usr/sbin/bonnie++ -f -s 100 -n 1 -r 0

What should I do next?  Should I capture some of the
tcpdump output or compile debugging or profiling into
the kernel for evaluation?

Thanks!

--- Trond Myklebust <trond.myklebust@fys.uio.no>
wrote:

> On Mon, 2007-05-21 at 07:18 -0700, John Frisk wrote:
> > Team,
> > I have been attempting to put together a linux HA
> NFS
> > server for my home environment.  I am currently
> > running two Debian etch 4.0 machines both running
> > vanilla 2.6.21.1 kernels that I compiled
> (exception
> > being the nvidia driver for video).  When I
> attempt to
> > do some performance benchmarks, a test of bonnie++
> > makes the NFS client stuck in a non-responsive
> > non-killable state.  The NFS server at this point
> is
> > not busy doing anything so I believe there may be
> some
> > condition leading to a deadlock in the client. 
> 
> Could you please try the patches
> 
>
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-002-no_congestion_wait_in_update_request.dif
>
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-003-cleanup_coalesce.dif
>
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-004-cleanup_coalesce2.dif
>
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-005-cleanup_readpages.dif
>
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-006-fix_dirtying_race.dif
>
http://client.linux-nfs.org/Linux-2.6.x/2.6.21/linux-2.6.21-007-fix_page_overflow.dif
> 
> and
> 
>
http://client.linux-nfs.org/Linux-2.6.x/2.6.22-rc2/linux-2.6.22-001-write_deadlock_avoidance.dif
> 
> That fixes all the write deadlock issues for me.
> 
> Trond
> 
> 



       
____________________________________________________________________________________Ready for the edge of your seat? 
Check out tonight's top picks on Yahoo! TV. 
http://tv.yahoo.com/

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs