processes stuck in D state

processes stuck in D state

[Pau Aliagas]

Since 2.2.4-ac28 and 2.4.3 I keep on getting processes in D state that I
cannot kill, usually mozilla or nautilus which use a large amount of RAM.
Today is galeon:

A ps -eo pid,stat,pcpu,nwchan,wchan=WIDE-WCHAN-COLUMN -o args shows the
following:
11520 D     0.0 105db1 down_write_failed /usr/bin/galeon-bin

This didn't happen neither with 2.4.2 nor with 2.4.3-pre7; I'm not sure
about pre8.

Pau

Re: processes stuck in D state

[Barry K. Nathan]

Pau Aliagas wrote:
> Since 2.2.4-ac28 and 2.4.3 I keep on getting processes in D state that I
> cannot kill, usually mozilla or nautilus which use a large amount of RAM.

I don't have time to help debug this, but I'm getting this too, with
2.4.3 final. The previous kernel I ran was 2.4.3-pre4, and it did not
have this problem.

In my case, it's usually mozilla (I'm seeing this with the daily
snapshots, but not with mozilla-0.8.1, at least not yet), but at least
once I saw it with freeamp (2.1rc5) too.

-Barry K. Nathan <barryn@pobox.com>

processes stuck in D state

[Zeev Fisher]

Hi!

I got a continuos problem of unkillable processes stuck in D state ( 
uninterruptable sleep ) on my Linux servers.
It happens randomly every time on other server on another process ( all 
the servers are configured the same with 2.4.18-10 kernel ). Here's an 
example :

root@lnx35 /]# ps -el|grep D
 F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD
000 D   911 29327     1  0  75   0    -  9382 lock_p ?        00:00:00 
calibre
000 D   894 30049 15854  0  75   0    -  8995 lock_p ?        00:00:01 
calibrewb
000 D   894 30092  8661  0  75   0    -  8995 lock_p ?        00:00:01 
calibrewb
000 D   894 29773 26052  0  75   0    -  8977 lock_p ?        00:00:01 
calibrewb


It was probably stuck while trying to get a lock (which was
certainly free) on an NFS volume mounted from a Netapp server.

Enabling debug mode on rpc ( echo '65535' >/proc/sys/sunrpc/rpc_debug ) 
didn't gave me any clue.
Tracing the stucked process pid doesn't give any output.

Those processes are there already few days and will stay there until 
next reboot.

The load average is now 4 ( although the machine is 100% idle ) and the 
system seems to work fine.
If other programs are started again they run and use the same mounts 
that the processes above are stuck on.

Another detail is that those problems started when i added the 'intr' 
option to my nfs mounted fs but i'm not sure. Also, i can't easily check 
that since this problem is not reproducible.

Has anyone noticed the same behavior ? Is this a well known problem ?


Thanks for your help.

-- 
Zeev Fisher - Unix System Administrator
Marvell Semiconductor Israel Ltd
Moshav Manof, D.N. Misgav 20184, ISRAEL
Email    -  Zeev.Fisher@il.marvell.com
Tel      -  + 972 4 9091402
Cell     -  + 972 54 995402
Fax      -  + 972 4 9091501
WWW Page:     http://www.marvell.com

------------------------------------------------------------------------
This message may contain confidential, proprietary or legally privileged
information. The information is intended only for the use of the individual
or entity named above. If the reader of this message is not the
intended recipient, you are hereby notified that any dissemination, distribution
or copying of this communication is strictly prohibited.
If you have received this communication in error, please notify us
immediately by telephone, or by e-mail and delete the message from your
computer.

Re: processes stuck in D state

[Michael Buesch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 05 May 2003 07:52, Zeev Fisher wrote:
> Hi!

Hi Zeev!

> I got a continuos problem of unkillable processes stuck in D state (
> uninterruptable sleep ) on my Linux servers.
> It happens randomly every time on other server on another process ( all
> the servers are configured the same with 2.4.18-10 kernel ). Here's an
> example :
[snip]
> Has anyone noticed the same behavior ? Is this a well known problem ?

I've had the same problem with some 2.4.21-preX twice (or maybe more times,
don't remember) on one of my machines.
IMHO it has something to do with NFS. (I'm using this box as a NFS-client).
I wish, I could reproduce it one more time, to do some traces, etc
on it. But I've not found a way to reproduce it, yet.

- -- 
Regards Michael Büsch
http://www.8ung.at/tuxsoft
 16:50:44 up 52 min,  1 user,  load average: 1.00, 1.00, 0.94
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+tnugoxoigfggmSgRAt8BAJ0deufnL/E6acpz4pIPZll8f48TIgCfWmcI
auSRmi6oyrTbqMVe+MrfuV4=
=ahIZ
-----END PGP SIGNATURE-----

Re: processes stuck in D state

[Mike Waychison]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN, Size: 1692 bytes --]



On Mon, 5 May 2003, Michael Buesch wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Monday 05 May 2003 07:52, Zeev Fisher wrote:
> > Hi!
>
> Hi Zeev!
>
> > I got a continuos problem of unkillable processes stuck in D state (
> > uninterruptable sleep ) on my Linux servers.
> > It happens randomly every time on other server on another process ( all
> > the servers are configured the same with 2.4.18-10 kernel ). Here's an
> > example :
> [snip]
> > Has anyone noticed the same behavior ? Is this a well known problem ?
>
> I've had the same problem with some 2.4.21-preX twice (or maybe more times,
> don't remember) on one of my machines.
> IMHO it has something to do with NFS. (I'm using this box as a NFS-client).
> I wish, I could reproduce it one more time, to do some traces, etc
> on it. But I've not found a way to reproduce it, yet.
>

This happens when you mount an NFS mount with the 'hard' option (default)
and a mount's handle expires incorrectly (eg: server crash).
Read the mount manpage for an explanation to the downsides of using
the 'soft' option.


Mike Waychison

> - --
> Regards Michael Büsch
> http://www.8ung.at/tuxsoft
>  16:50:44 up 52 min,  1 user,  load average: 1.00, 1.00, 0.94
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.1 (GNU/Linux)
>
> iD8DBQE+tnugoxoigfggmSgRAt8BAJ0deufnL/E6acpz4pIPZll8f48TIgCfWmcI
> auSRmi6oyrTbqMVe+MrfuV4=
> =ahIZ
> -----END PGP SIGNATURE-----
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: processes stuck in D state

[Michael Buesch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 05 May 2003 17:24, Mike Waychison wrote:
> On Mon, 5 May 2003, Michael Buesch wrote:
> > On Monday 05 May 2003 07:52, Zeev Fisher wrote:
> > > Hi!
> >
> > Hi Zeev!
> >
> > > I got a continuos problem of unkillable processes stuck in D state (
> > > uninterruptable sleep ) on my Linux servers.
> > > It happens randomly every time on other server on another process ( all
> > > the servers are configured the same with 2.4.18-10 kernel ). Here's an
> > > example :
> >
> > [snip]
> >
> > > Has anyone noticed the same behavior ? Is this a well known problem ?
> >
> > I've had the same problem with some 2.4.21-preX twice (or maybe more
> > times, don't remember) on one of my machines.
> > IMHO it has something to do with NFS. (I'm using this box as a
> > NFS-client). I wish, I could reproduce it one more time, to do some
> > traces, etc on it. But I've not found a way to reproduce it, yet.
>
> This happens when you mount an NFS mount with the 'hard' option (default)
> and a mount's handle expires incorrectly (eg: server crash).
> Read the mount manpage for an explanation to the downsides of using
> the 'soft' option.
>
>
> Mike Waychison

my fstab-entry:
192.168.0.50:/mnt/nfs_1 /mnt/nfs_1      nfs             rw,hard,intr,user,nodev,nosuid,exec         0 0

from man mount:
[snip] The process cannot be interrupted or killed unless you also specify intr. [/snip]

I can't interrupt any process that accessed the NFS-server
while shutting down the server, although intr is specified.
_That's_ my problem. :)

- -- 
Regards Michael Büsch
http://www.8ung.at/tuxsoft
 18:23:58 up 48 min,  3 users,  load average: 1.20, 1.05, 0.93
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+tpCZoxoigfggmSgRAkmdAJwM/L8mZpS+DE2WzjzrXuRdxuY98QCgin1l
aKik6/WGFwWXMjd8pjwHIXw=
=akJd
-----END PGP SIGNATURE-----

Re: processes stuck in D state

[jw schultz]

On Mon, May 05, 2003 at 06:25:48PM +0200, Michael Buesch wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Monday 05 May 2003 17:24, Mike Waychison wrote:
> > On Mon, 5 May 2003, Michael Buesch wrote:
> > > On Monday 05 May 2003 07:52, Zeev Fisher wrote:
> > > > Hi!
> > >
> > > Hi Zeev!
> > >
> > > > I got a continuos problem of unkillable processes stuck in D state (
> > > > uninterruptable sleep ) on my Linux servers.
> > > > It happens randomly every time on other server on another process ( all
> > > > the servers are configured the same with 2.4.18-10 kernel ). Here's an
> > > > example :
> > >
> > > [snip]
> > >
> > > > Has anyone noticed the same behavior ? Is this a well known problem ?
> > >
> > > I've had the same problem with some 2.4.21-preX twice (or maybe more
> > > times, don't remember) on one of my machines.
> > > IMHO it has something to do with NFS. (I'm using this box as a
> > > NFS-client). I wish, I could reproduce it one more time, to do some
> > > traces, etc on it. But I've not found a way to reproduce it, yet.
> >
> > This happens when you mount an NFS mount with the 'hard' option (default)
> > and a mount's handle expires incorrectly (eg: server crash).
> > Read the mount manpage for an explanation to the downsides of using
> > the 'soft' option.
> >
> >
> > Mike Waychison
> 
> my fstab-entry:
> 192.168.0.50:/mnt/nfs_1 /mnt/nfs_1      nfs             rw,hard,intr,user,nodev,nosuid,exec         0 0
> 
> from man mount:
> [snip] The process cannot be interrupted or killed unless you also specify intr. [/snip]
> 
> I can't interrupt any process that accessed the NFS-server
> while shutting down the server, although intr is specified.
> _That's_ my problem. :)

I had a similar problem with SuSE's 2.4.18.  Random processes
seemed to go into D state from whence intr is useless.

I rebuilt the kernel with NFSv3 disabled and that problem
went away.  The logs are full of
    May  5 14:54:15 duncan kernel: NFS: NFSv3 not supported.
    May  5 14:54:15 duncan kernel: nfs warning: mount version older than kernel
but that i can live with.  Processes hung and umount failing
i cannot abide.

If there is a better answer, i'm listening.


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

processes stuck in D state

[Michael Buesch]

=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi!

Please take a look at this problem:

[linux-kernel-mailing-list thread]
http://marc.theaimsgroup.com/?t=3D98639966100003&r=3D1&w=3D2

thanks.
Please cc me, as I'm not subscribed to the nfs-list.

=2D --=20
Regards Michael B=FCsch
http://www.8ung.at/tuxsoft
 16:28:34 up 20 min,  1 user,  load average: 1.07, 0.97, 0.66
=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+t8wdoxoigfggmSgRAm5aAJsGJLPe9yUd4sqah4yiU0GsMIAGzACfSa2+
gAMZvSHQirHmE8yZChpgH/8=3D
=3Dpka2
=2D----END PGP SIGNATURE-----



-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: processes stuck in D state

[Lever, Charles]

hi michael-

i'm not sure why you mailed neilb -- this appears to
be NFS client related, not server related.

can you spell out the sequence of events that leads
to the stuck processes?  it looks like the client
is working-as-designed, but if you can provide more
details, we can verify what's going on.

> -----Original Message-----
> From: Michael Buesch [mailto:fsdeveloper@yahoo.de]
> Sent: Tuesday, May 06, 2003 10:52 AM
> To: neilb@cse.unsw.edu.au
> Cc: nfs@lists.sourceforge.net
> Subject: [NFS] processes stuck in D state
>=20
>=20
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>=20
> Hi!
>=20
> Please take a look at this problem:
>=20
> [linux-kernel-mailing-list thread]
> http://marc.theaimsgroup.com/?t=3D98639966100003&r=3D1&w=3D2
>=20
> thanks.
> Please cc me, as I'm not subscribed to the nfs-list.
>=20
> - --=20
> Regards Michael B=FCsch
> http://www.8ung.at/tuxsoft
>  16:28:34 up 20 min,  1 user,  load average: 1.07, 0.97, 0.66
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.1 (GNU/Linux)
>=20
> iD8DBQE+t8wdoxoigfggmSgRAm5aAJsGJLPe9yUd4sqah4yiU0GsMIAGzACfSa2+
> gAMZvSHQirHmE8yZChpgH/8=3D
> =3Dpka2
> -----END PGP SIGNATURE-----
>=20
>=20
>=20
> -------------------------------------------------------
> Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
> The only event dedicated to issues related to Linux=20
> enterprise solutions
> www.enterpriselinuxforum.com
>=20
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
>=20


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: processes stuck in D state

[Trond Myklebust]

>>>>> " " == Michael Buesch <fsdeveloper@yahoo.de> writes:

     > Hi!  Please take a look at this problem:

     > [linux-kernel-mailing-list thread]
     > http://marc.theaimsgroup.com/?t=98639966100003&r=1&w=2

If I can hazard a guess: someone is firewalling the lockd port and/or
the statd port.

Either mount using the 'nolock' option, or fix the firewall (see the
HOWTO and/or FAQ).

Cheers,
  Trond


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: processes stuck in D state

[Michael Buesch]

=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 06 May 2003 17:20, Trond Myklebust wrote:
> >>>>> " " =3D=3D Michael Buesch <fsdeveloper@yahoo.de> writes:
>      > Hi!  Please take a look at this problem:
>      >
>      > [linux-kernel-mailing-list thread]
>      > http://marc.theaimsgroup.com/?t=3D98639966100003&r=3D1&w=3D2
>
> If I can hazard a guess: someone is firewalling the lockd port and/or
> the statd port.
>
> Either mount using the 'nolock' option, or fix the firewall (see the
> HOWTO and/or FAQ).

To reproduce the problem:
=2D - mount some nfs from a server in your lan.
=2D - Open an app, that uses the mounted fs. I've simply opened a
  konqueror-window for the directory where the nfs is mounted.
=2D - shut down or crash the server or just pull the network-cable.
=2D - Now the konqueror-process is nonkillable in D state. There's no
  chance to kill it.

I've tried it with all firewalls disabled, but the problem resists.

> Cheers,
>   Trond

@linux-kernel-mailing-list: I've posted a thread to nfs-mailing list with
the same topic as in lkml. IMHO this is the better list for this problem. :)

=2D --=20
Regards Michael B=FCsch
http://www.8ung.at/tuxsoft
 17:34:35 up  1:26,  5 users,  load average: 1.52, 1.32, 1.13
=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+t9fWoxoigfggmSgRAvrcAJ4i+i3V+kcRd+kLHS7cb2WDZDHKsQCfWljd
rwtAFK4ONkJHzVck03t7F5U=3D
=3DgHuP
=2D----END PGP SIGNATURE-----



-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [NFS] processes stuck in D state

[Michael Buesch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 06 May 2003 17:20, Trond Myklebust wrote:
> >>>>> " " == Michael Buesch <fsdeveloper@yahoo.de> writes:
>      > Hi!  Please take a look at this problem:
>      >
>      > [linux-kernel-mailing-list thread]
>      > http://marc.theaimsgroup.com/?t=98639966100003&r=1&w=2
>
> If I can hazard a guess: someone is firewalling the lockd port and/or
> the statd port.
>
> Either mount using the 'nolock' option, or fix the firewall (see the
> HOWTO and/or FAQ).

To reproduce the problem:
- - mount some nfs from a server in your lan.
- - Open an app, that uses the mounted fs. I've simply opened a
  konqueror-window for the directory where the nfs is mounted.
- - shut down or crash the server or just pull the network-cable.
- - Now the konqueror-process is nonkillable in D state. There's no
  chance to kill it.

I've tried it with all firewalls disabled, but the problem resists.

> Cheers,
>   Trond

@linux-kernel-mailing-list: I've posted a thread to nfs-mailing list with
the same topic as in lkml. IMHO this is the better list for this problem. :)

- -- 
Regards Michael Büsch
http://www.8ung.at/tuxsoft
 17:34:35 up  1:26,  5 users,  load average: 1.52, 1.32, 1.13
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+t9fWoxoigfggmSgRAvrcAJ4i+i3V+kcRd+kLHS7cb2WDZDHKsQCfWljd
rwtAFK4ONkJHzVck03t7F5U=
=gHuP
-----END PGP SIGNATURE-----

RE: processes stuck in D state

[Lever, Charles]

> To reproduce the problem:
> - - mount some nfs from a server in your lan.
> - - Open an app, that uses the mounted fs. I've simply opened a
>   konqueror-window for the directory where the nfs is mounted.
> - - shut down or crash the server or just pull the network-cable.
> - - Now the konqueror-process is nonkillable in D state. There's no
>   chance to kill it.

does the problem persist after you reconnect the network cable?
what happens when the server becomes available again?
are you mounting with UDP or TCP?


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: processes stuck in D state

[Trond Myklebust]

>>>>> " " == Michael Buesch <fsdeveloper@yahoo.de> writes:


     > To reproduce the problem:
     > - - mount some nfs from a server in your lan.
     > - - Open an app, that uses the mounted fs. I've simply opened a
     >   konqueror-window for the directory where the nfs is mounted.
     > - - shut down or crash the server or just pull the
     >     network-cable.
     > - - Now the konqueror-process is nonkillable in D
     >     state. There's no
     >   chance to kill it.

Unless you are using the 'intr' or 'soft' mount flags, then that is
*documented and expected* behaviour.

It is true that even when using the 'intr' mount flag, you don't
always succeed in killing a task that is hanging on NFS. That is
usually due to the fact that it is waiting on some semaphore that is
held by another process. semaphores always sleep in the
TASK_UNINTERRUPTIBLE state, so they cannot be signalled.
Linus has suggested a solution to this problem: to set up a special
class of semaphores that are killable with 'SIGKILL', but doing that
(and then replacing all those semaphores in the VFS and VM) is not
going to happen before 2.7.x. at the earliest.

However, as I've mentioned on this list *many* times before: there
exists a workaround if you are wanting to kill all processes in order
to unmount the partition:
  kill -9 all the processes.
  kill -9 rpciod.

Cheers,
  Trond


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [NFS] processes stuck in D state

[Trond Myklebust]

>>>>> " " == Michael Buesch <fsdeveloper@yahoo.de> writes:


     > To reproduce the problem:
     > - - mount some nfs from a server in your lan.
     > - - Open an app, that uses the mounted fs. I've simply opened a
     >   konqueror-window for the directory where the nfs is mounted.
     > - - shut down or crash the server or just pull the
     >     network-cable.
     > - - Now the konqueror-process is nonkillable in D
     >     state. There's no
     >   chance to kill it.

Unless you are using the 'intr' or 'soft' mount flags, then that is
*documented and expected* behaviour.

It is true that even when using the 'intr' mount flag, you don't
always succeed in killing a task that is hanging on NFS. That is
usually due to the fact that it is waiting on some semaphore that is
held by another process. semaphores always sleep in the
TASK_UNINTERRUPTIBLE state, so they cannot be signalled.
Linus has suggested a solution to this problem: to set up a special
class of semaphores that are killable with 'SIGKILL', but doing that
(and then replacing all those semaphores in the VFS and VM) is not
going to happen before 2.7.x. at the earliest.

However, as I've mentioned on this list *many* times before: there
exists a workaround if you are wanting to kill all processes in order
to unmount the partition:
  kill -9 all the processes.
  kill -9 rpciod.

Cheers,
  Trond

Re: processes stuck in D state

[Michael Buesch]

=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 06 May 2003 18:05, Trond Myklebust wrote:
> >>>>> " " =3D=3D Michael Buesch <fsdeveloper@yahoo.de> writes:
>      > To reproduce the problem:
>      > - - mount some nfs from a server in your lan.
>      > - - Open an app, that uses the mounted fs. I've simply opened a
>      >   konqueror-window for the directory where the nfs is mounted.
>      > - - shut down or crash the server or just pull the
>      >     network-cable.
>      > - - Now the konqueror-process is nonkillable in D
>      >     state. There's no
>      >   chance to kill it.
>
> Unless you are using the 'intr' or 'soft' mount flags, then that is
> *documented and expected* behaviour.

I'm using intr.

> However, as I've mentioned on this list *many* times before: there
> exists a workaround if you are wanting to kill all processes in order
> to unmount the partition:
>   kill -9 all the processes.
>   kill -9 rpciod.

kill -9 doesn't work for me to kill the app.

=2D --=20
Regards Michael B=FCsch
http://www.8ung.at/tuxsoft
 18:28:55 up  2:20,  5 users,  load average: 1.02, 1.06, 1.06
=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+t+MhoxoigfggmSgRAkeqAJ0c71DxLZ13/CHqUXlTa8TvjAt2iwCeLO34
s7crt56Gr8JyKxCLZMbrNvc=3D
=3Dz8EU
=2D----END PGP SIGNATURE-----



-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [NFS] processes stuck in D state

[Michael Buesch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 06 May 2003 18:05, Trond Myklebust wrote:
> >>>>> " " == Michael Buesch <fsdeveloper@yahoo.de> writes:
>      > To reproduce the problem:
>      > - - mount some nfs from a server in your lan.
>      > - - Open an app, that uses the mounted fs. I've simply opened a
>      >   konqueror-window for the directory where the nfs is mounted.
>      > - - shut down or crash the server or just pull the
>      >     network-cable.
>      > - - Now the konqueror-process is nonkillable in D
>      >     state. There's no
>      >   chance to kill it.
>
> Unless you are using the 'intr' or 'soft' mount flags, then that is
> *documented and expected* behaviour.

I'm using intr.

> However, as I've mentioned on this list *many* times before: there
> exists a workaround if you are wanting to kill all processes in order
> to unmount the partition:
>   kill -9 all the processes.
>   kill -9 rpciod.

kill -9 doesn't work for me to kill the app.

- -- 
Regards Michael Büsch
http://www.8ung.at/tuxsoft
 18:28:55 up  2:20,  5 users,  load average: 1.02, 1.06, 1.06
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+t+MhoxoigfggmSgRAkeqAJ0c71DxLZ13/CHqUXlTa8TvjAt2iwCeLO34
s7crt56Gr8JyKxCLZMbrNvc=
=z8EU
-----END PGP SIGNATURE-----

Re: processes stuck in D state

[Trond Myklebust]

>>>>> " " == Michael Buesch <fsdeveloper@yahoo.de> writes:

    >> kill -9 all the processes.  kill -9 rpciod.

     > kill -9 doesn't work for me to kill the app.

I didn't say kill the app. I said signal it with -9, then signal
rpciod.

Cheers,
  Trond


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [NFS] processes stuck in D state

[Trond Myklebust]

>>>>> " " == Michael Buesch <fsdeveloper@yahoo.de> writes:

    >> kill -9 all the processes.  kill -9 rpciod.

     > kill -9 doesn't work for me to kill the app.

I didn't say kill the app. I said signal it with -9, then signal
rpciod.

Cheers,
  Trond

RE: processes stuck in D state

[pwitting]

Actually, I have seen this on the server before, specifically
When using an older version of IBM's JFS for Linux with RH 7.3

I worked with the JFS team and JFS v1.1.0 and kernel 2.4.20 
seemed to stabilize it, at least I've seen no more "freezes"
as a result. Judging by your mail address this might be your 
problem as well.

One noticeable symptom is that cd'ing to an affected dir and 
attempting an ls would also freeze (I had a large (20GB+) file 
copy going, so I usually knew what the affected dir was.

Two other things that helped:
1) increasing the # of nfs threads (120 or more)
2) ensuring the uid/gid the remote thread was using existed on 
the server. (sounds stupid but it helped)

neither "cured" the issue, but it went from being reproducible 
to being occasional.

Good Luck.

> From: "Lever, Charles" <Charles.Lever@netapp.com>
>
> i'm not sure why you mailed neilb -- this appears to
> be NFS client related, not server related.
> 
> can you spell out the sequence of events that leads
> to the stuck processes?  it looks like the client
> is working-as-designed, but if you can provide more
> details, we can verify what's going on.
> 
>> -----Original Message-----
>> From: Michael Buesch [mailto:fsdeveloper@yahoo.de]
>> Sent: Tuesday, May 06, 2003 10:52 AM
>> To: neilb@cse.unsw.edu.au
>> Cc: nfs@lists.sourceforge.net
>> Subject: [NFS] processes stuck in D state
>>
>> Please take a look at this problem:
>>
>> [linux-kernel-mailing-list thread]
>> http://marc.theaimsgroup.com/?t=3D98639966100003&r=3D1&w=3D2
>>
>> thanks.
>> Please cc me, as I'm not subscribed to the nfs-list.



-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: processes stuck in D state

[Michael Buesch]

=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 06 May 2003 19:09, pwitting@Cyveillance.com wrote:
> Actually, I have seen this on the server before, specifically
> When using an older version of IBM's JFS for Linux with RH 7.3

I'm running ext3 on the server.

> I worked with the JFS team and JFS v1.1.0 and kernel 2.4.20
> seemed to stabilize it, at least I've seen no more "freezes"
> as a result.

server-kernel is 2.4.21-pre6.

> Judging by your mail address this might be your
> problem as well.

You refer to the "fs" in my e-mail address? :)
The "fs" stands for "FreeSoftware".

> One noticeable symptom is that cd'ing to an affected dir and
> attempting an ls would also freeze (I had a large (20GB+) file
> copy going, so I usually knew what the affected dir was.
>
> Two other things that helped:
> 1) increasing the # of nfs threads (120 or more)
> 2) ensuring the uid/gid the remote thread was using existed on
> the server. (sounds stupid but it helped)
>
> neither "cured" the issue, but it went from being reproducible
> to being occasional.
>
> Good Luck.

thanks. :)

=2D --=20
Regards Michael B=FCsch
http://www.8ung.at/tuxsoft
 19:26:58 up  3:18,  2 users,  load average: 1.01, 1.08, 1.05
=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+t/FhoxoigfggmSgRAqgLAJ9hGo5l2c1y/oiZxamh/M88avZYIwCggzTc
lgyMmTHSGnxK17Xfq1Vb/eg=3D
=3DNi5P
=2D----END PGP SIGNATURE-----



-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [NFS] processes stuck in D state

[Michael Buesch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 06 May 2003 18:54, Trond Myklebust wrote:
> >>>>> " " =3D=3D Michael Buesch <fsdeveloper@yahoo.de> writes:
>     >> kill -9 all the processes.  kill -9 rpciod.
>     >>
>      > kill -9 doesn't work for me to kill the app.
>
> I didn't say kill the app. I said signal it with -9, then signal
> rpciod.

Ah, I understand. :)

> Cheers,
>   Trond

- --=20
Regards Michael B=FCsch
http://www.8ung.at/tuxsoft
 19:31:20 up  3:22,  2 users,  load average: 1.23, 1.09, 1.04
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+t/G3oxoigfggmSgRAq5BAJ0SezM+y1LFnwglArReHERXb2VJZQCeKKd0
Sx6RqCkOvm4FvgTCVyx2gCE=3D
=3DK8c7
-----END PGP SIGNATURE-----

Re: [NFS] processes stuck in D state

[Michael Buesch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 06 May 2003 18:54, Trond Myklebust wrote:
> >>>>> " " == Michael Buesch <fsdeveloper@yahoo.de> writes:
>     >> kill -9 all the processes.  kill -9 rpciod.
>     >>
>      > kill -9 doesn't work for me to kill the app.
>
> I didn't say kill the app. I said signal it with -9, then signal
> rpciod.

Ah, I understand. :)

> Cheers,
>   Trond

- -- 
Regards Michael Büsch
http://www.8ung.at/tuxsoft
 19:31:20 up  3:22,  2 users,  load average: 1.23, 1.09, 1.04
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+t/G3oxoigfggmSgRAq5BAJ0SezM+y1LFnwglArReHERXb2VJZQCeKKd0
Sx6RqCkOvm4FvgTCVyx2gCE=
=K8c7
-----END PGP SIGNATURE-----

RE: processes stuck in D state

[Guolin Cheng]

Hi, all,

 We are encountering the same problem here as well,=20
=20
[root@arc100 root]# ps auxw
......
 andy      9087  0.0  3.2 12900 10312 ?       D    May05   0:02
/0/tmp/av_explore .index /net/arc295/0/DQ_crawl26.20030416034937.arc.gz
......

although the nfs server has no problem at all. If I try to "ls
/net/arc295/0" then the new "ls" process will hang as well.
The method I followed to fix the problem is:

[root@arc100 root]# umount -f /net/arc295/0
umount2: Device or resource busy
umount: /net/arc295/0: Illegal seek
[root@arc100 root]# /etc/init.d/amd restart
Stopping amd:                                              [  OK  ]
Starting amd:                                              [  OK  ]
[root@arc100 root]# cd /net/arc295/0
[root@arc100 0]# ls=20

 My nfs clients/servers has the same configurations:=20

	Redhat 8.0
 	General Linux Kernel 2.4.20 ("nfs over tcp" is enabled)
	gcc-3.2-7=20
 	amd ( am-utils-6.0.7-9 )
	amd mount options in map amd.master
(opts:=3Drw,intr,nfsv3,tcp,nosuid,nodev,noresvport)
      The real amd mount status for the nfs directory:
        arc295:/0 on /.amd_mnt/arc295/host/0 type nfs
(rw,intr,nfsv3,tcp,nosuid,nodev,noresvport,dev=3D0000f10e,vers=3D3,proto=
=3Dtcp)

 Any one need more info the shoot the sort of problem, let me know.

 Thanks.
 --Guolin Cheng


-----Original Message-----
From: Lever, Charles [mailto:Charles.Lever@netapp.com]
Sent: Tuesday, May 06, 2003 8:18 AM
To: Michael Buesch
Cc: nfs@lists.sourceforge.net; neilb@cse.unsw.edu.au
Subject: RE: [NFS] processes stuck in D state


hi michael-

i'm not sure why you mailed neilb -- this appears to
be NFS client related, not server related.

can you spell out the sequence of events that leads
to the stuck processes?  it looks like the client
is working-as-designed, but if you can provide more
details, we can verify what's going on.

> -----Original Message-----
> From: Michael Buesch [mailto:fsdeveloper@yahoo.de]
> Sent: Tuesday, May 06, 2003 10:52 AM
> To: neilb@cse.unsw.edu.au
> Cc: nfs@lists.sourceforge.net
> Subject: [NFS] processes stuck in D state
>=20
>=20
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>=20
> Hi!
>=20
> Please take a look at this problem:
>=20
> [linux-kernel-mailing-list thread]
> http://marc.theaimsgroup.com/?t=3D98639966100003&r=3D1&w=3D2
>=20
> thanks.
> Please cc me, as I'm not subscribed to the nfs-list.
>=20
> - --=20
> Regards Michael B=FCsch
> http://www.8ung.at/tuxsoft
>  16:28:34 up 20 min,  1 user,  load average: 1.07, 0.97, 0.66
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.1 (GNU/Linux)
>=20
> iD8DBQE+t8wdoxoigfggmSgRAm5aAJsGJLPe9yUd4sqah4yiU0GsMIAGzACfSa2+
> gAMZvSHQirHmE8yZChpgH/8=3D
> =3Dpka2
> -----END PGP SIGNATURE-----
>=20
>=20
>=20
> -------------------------------------------------------
> Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
> The only event dedicated to issues related to Linux=20
> enterprise solutions
> www.enterpriselinuxforum.com
>=20
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
>=20


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise =
solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Processes stuck in D state

[Olaf Kirch]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]


Hi,

I spent much of today investigating a weird NFS problem on 2.6.3.

After one of our servers went away and came back, several processes on
a ppc machine were left in D state. They did not get woken up during
the whole day, until I did a "umount -f" after several hours of debugging.

The internal state of this RPC client looks a little weird. I'm
attaching some debug output that shows where they got stuck. Some
general observation:

 -	this does not seem a queue corruption bug, which is good :)
 -	the tasks were sleeping on different wait queues (pending,
 	sending, 1 one resend)
 -	all tasks have a tk_timeout value of 0
 -	the ntimeo values of the RTT estimators being 0 looks a
	little weird, given that the mount froze because the
	server wasn't reachable.
 -	the task on the resend queue has a timer with
	tk_timer.expires != 0, but unfortunately I forgot to check whether
	it was active.	But I doubt it; I had debugging enabled for much
	of the day and the tk_pid in question never showed up in the log

I'm not sure yet what exactly happened here. I don't understand how a
task on xprt->pending can have a timeout value of 0...

Does anyone have an idea what might be going wrong here?

Olaf
-- 
Olaf Kirch     |  Stop wasting entropy - start using predictable
okir@suse.de   |  tempfile names today!
---------------+ 

[-- Attachment #2: nfs-messages --]
[-- Type: text/plain, Size: 1489 bytes --]

 Found NFS mount, server=Hilbert2,v3,rsize=8192,wsize=8192
   RPC client 6 users
     Active RPC tasks for this client:
       task 21384, status=-11, timeout=0, active, sleeping, on queue c45ea044(xprt_sending)
       task 9969, status=-11, timeout=0, active, sleeping, on queue c45ea044(xprt_sending)
       task 52431, status=-11, timeout=0, active, sleeping, on queue c45ea044(xprt_sending)
       task 43528, status=-11, timeout=0, active, sleeping, on queue c45ea044(xprt_sending)
       task 43527, status=-11, timeout=0, timer, async, active, sleeping, on queue c45ea050(xprt_resend)
       task 55816, status=0, timeout=0, active, sleeping, on queue c45ea05c(xprt_pending)
     Transport c45ea000, sockstate=0x1
       cong 256/cwnd 256
       RTT estimates (def timeout 700):
         0: rtt   15 srtt  100 ntimeo   0
         1: rtt   15 srtt  100 ntimeo   0
         2: rtt   49 srtt  100 ntimeo   0
         3: rtt   15 srtt  100 ntimeo   0
         4: rtt   15 srtt  100 ntimeo   0
     RPC wait queue sending:
       task 43528, status=-11, timeout=0, active, sleeping
       task 52431, status=-11, timeout=0, active, sleeping
       task 9969, status=-11, timeout=0, active, sleeping
       task 21384, status=-11, timeout=0, active, sleeping
     RPC wait queue pending:
       task 55816, status=0, timeout=0, active, sleeping
     RPC wait queue resend:
       task 43527, status=-11, timeout=0, active timer, async, active, sleeping
     RPC wait queue backlog: empty