Client Lockups

Client Lockups

[IHE Lists]

Here's the situation:

We have a multithreaded Java application that does a great deal of IO
over NFS. Almost daily, sometimes several times within a single day,
we get an application deadlock. The error in the kernel syslog message
is:

Dec 11 11:09:05 <hostname> kernel: RPC: error 512 connecting to server
<NFS server>

This error repeats numerous times.

The client mount options are:

rw,hard,intr,rsize=8192,wsize=8192,noatime,nfsvers=3,tcp,timeo=600,retrans=2

We have a deadlock monitor that attempts to restart the application by
first sending the process a "kill -9", then running the init.d script.
Of course, the threads stuck in the D state do not die, so the restart
fails. This cycle continues for almost 25 minutes (almost
consistently) before it succeeds.

I believe the eventual total deadlock is a problem in the Java virtual
machine, but I cannot determine what is causing the problem to begin
with. Other applications on the same system do not lockup accessing
other NFS mounts.

I've seen this problem on a custom kernel running 2.4.20 with the -aa1
patches and all the other applicable 2.4.20 patches from Trond and
Neil. I've also seen it occur on SLES8 kernels 2.4.21-241, 2.4.21-251,
and 2.4.21-261. Our systems are somewhat custom, so they are all
running nfs-utils 1.0.6, but the base is currently Red Hat 7.3.

I guess my primary question is: this a kernel problem, an nfs-utils
problem, or does the problem lie somewhere else?

Any help is greatly appreciated!

-Brian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Client Lockups

[Trond Myklebust]

m=E5 den 13.12.2004 Klokka 00:08 (-0600) skreiv IHE Lists:
> Here's the situation:
>=20
> We have a multithreaded Java application that does a great deal of IO
> over NFS. Almost daily, sometimes several times within a single day,
> we get an application deadlock. The error in the kernel syslog message
> is:
>=20
> Dec 11 11:09:05 <hostname> kernel: RPC: error 512 connecting to server
> <NFS server>

error 512 =3D=3D ERESTARTSYS. That just means that someone interrupted the
RPC call: your deadlock monitor would trigger this error when it sends
the kill -9.

> I believe the eventual total deadlock is a problem in the Java virtual
> machine, but I cannot determine what is causing the problem to begin
> with. Other applications on the same system do not lockup accessing
> other NFS mounts.

Have you tried using "strace" to figure out which system call is
hanging?

Also, which server are you using? The basic RedHat 7.3 certainly would
not support tcp on the server side.

Cheers,
  Trond

--=20
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs