automount in status D

automount in status D

[Carsten Aulbert]

Hi all,

we have a weird problem here and I fear I need to reboot the box
(painful, many users involved), but maybe there is a way out:

I'm automounting many nodes and usually everything seems to work nicely.
However, I'm seeing this here right now:

# ps aux|grep auto.node
root      2022  0.0  0.0  30304  1240 ?        Ds   Jun19   0:21 /usr/sbin/automount --pid-file=/var/run/autofs/_atlas_node.pid --timeout=5 /atlas/node yp auto.node_local
root     11489  0.0  0.0  30516  1108 ?        D    Jun26   0:00 /usr/sbin/automount --pid-file=/var/run/autofs/_atlas_node.pid --timeout=5 /atlas/node yp auto.node_local
root     28610  0.0  0.0   2736   656 pts/31   S+   10:03   0:00 grep auto.node


As you can see there are two automounter processes in D (or Ds) state sitting there for the same mount point (the PID-file contains 2022). 

My questions:

Any idea how this happened?

Any idea how to get this resolved (possibly without a reboot)?

TIA

Carsten

PS: 
# ypcat auto.node_local
-fstype=nfs,nfsvers=3,soft,intr,rsize=32768,wsize=32768,tcp       &:/local

Re: automount in status D

[Ian Kent]

On Fri, 2008-06-27 at 10:05 +0200, Carsten Aulbert wrote:
> Hi all,
> 
> we have a weird problem here and I fear I need to reboot the box
> (painful, many users involved), but maybe there is a way out:
> 
> I'm automounting many nodes and usually everything seems to work nicely.
> However, I'm seeing this here right now:
> 
> # ps aux|grep auto.node
> root      2022  0.0  0.0  30304  1240 ?        Ds   Jun19   0:21 /usr/sbin/automount --pid-file=/var/run/autofs/_atlas_node.pid --timeout=5 /atlas/node yp auto.node_local
> root     11489  0.0  0.0  30516  1108 ?        D    Jun26   0:00 /usr/sbin/automount --pid-file=/var/run/autofs/_atlas_node.pid --timeout=5 /atlas/node yp auto.node_local
> root     28610  0.0  0.0   2736   656 pts/31   S+   10:03   0:00 grep auto.node
> 
> 
> As you can see there are two automounter processes in D (or Ds) state sitting there for the same mount point (the PID-file contains 2022). 
> 
> My questions:
> 
> Any idea how this happened?

It's been a long time since I've seen this reported.
What distro and kernel?

> 
> Any idea how to get this resolved (possibly without a reboot)?

There is nothing you can do if the process is stuck in an
uninterruptible state in the kernel.

Ian

Re: automount in status D

[Carsten Aulbert]

Ian Kent wrote:

>> Any idea how this happened?
> 
> It's been a long time since I've seen this reported.
> What distro and kernel?
> 

Sorry, Debian Etch with self compiled kernel:
h2:~# uname -a
Linux h2 2.6.24.4-storage #1 SMP Sun Apr 20 09:33:42 CEST 2008 x86_64
x86_64 x86_64 GNU/Linux
h2:~# apt-cache show autofs|grep Version
Version: 4.1.4-13

>> Any idea how to get this resolved (possibly without a reboot)?
> 
> There is nothing you can do if the process is stuck in an
> uninterruptible state in the kernel.

Well, as bad as I feared it to be :(

Thanks already, reboot is already announced to the users. Anything I
might help you with to corner that bug?

Cheers

Carsten

-- 
Dr. Carsten Aulbert - Max Planck Institut für Gravitationsphysik
Callinstraße 38, 30167 Hannover, Germany
Fon: +49 511 762 17185, Fax: +49 511 762 17193
http://www.top500.org/system/9234 | http://www.top500.org/connfam/6/list/31

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: automount in status D

[Ian Kent]

On Fri, 2008-06-27 at 13:35 +0200, Carsten Aulbert wrote:
> 
> Ian Kent wrote:
> 
> >> Any idea how this happened?
> > 
> > It's been a long time since I've seen this reported.
> > What distro and kernel?
> > 
> 
> Sorry, Debian Etch with self compiled kernel:
> h2:~# uname -a
> Linux h2 2.6.24.4-storage #1 SMP Sun Apr 20 09:33:42 CEST 2008 x86_64
> x86_64 x86_64 GNU/Linux
> h2:~# apt-cache show autofs|grep Version
> Version: 4.1.4-13
> 
> >> Any idea how to get this resolved (possibly without a reboot)?
> > 
> > There is nothing you can do if the process is stuck in an
> > uninterruptible state in the kernel.
> 
> Well, as bad as I feared it to be :(
> 
> Thanks already, reboot is already announced to the users. Anything I
> might help you with to corner that bug?

Aggh .. it's been soooo long since I've heard of this happening I can't
remember.

One thing that is interesting is that the autofs4 kernel module doesn't
do an uninterruptible sleep "anywhere". It hasn't since long before
2.6.24. So if user space can't recover by doing a control-c (maybe even
a couple of times) then it's possibly not actually autofs.

Ian

Re: automount in status D

[Carsten Aulbert]

Hi Ian,

Ian Kent wrote:

> One thing that is interesting is that the autofs4 kernel module doesn't
> do an uninterruptible sleep "anywhere". It hasn't since long before
> 2.6.24. So if user space can't recover by doing a control-c (maybe even
> a couple of times) then it's possibly not actually autofs.

Well, when I tried to umount it manually (with options -f -l -i) stopped
indefinitely when doing a readlink on the mount point itself and never
returned from its own account.

I was able to CTRL-C this one but not getting the mount point removed. A
nasty side effect was that df never returned either.

I did a bit more log reading and it seems that the node got a mount as
well as a umount request yesterday evening and from the nodes
perspective everything looked fine.

In the end I guess this falls into the category of: Hopefully it won't
happen again too soon ;)

If you come up with an idea, please let me know. I'll stay on the list,
reading quietly.

Cheers

Carsten

Re: automount in status D

[Jeff Moyer]

Ian Kent <raven@themaw.net> writes:

> On Fri, 2008-06-27 at 13:35 +0200, Carsten Aulbert wrote:
>> 
>> Ian Kent wrote:
>> 
>> >> Any idea how this happened?
>> > 
>> > It's been a long time since I've seen this reported.
>> > What distro and kernel?
>> > 
>> 
>> Sorry, Debian Etch with self compiled kernel:
>> h2:~# uname -a
>> Linux h2 2.6.24.4-storage #1 SMP Sun Apr 20 09:33:42 CEST 2008 x86_64
>> x86_64 x86_64 GNU/Linux
>> h2:~# apt-cache show autofs|grep Version
>> Version: 4.1.4-13
>> 
>> >> Any idea how to get this resolved (possibly without a reboot)?
>> > 
>> > There is nothing you can do if the process is stuck in an
>> > uninterruptible state in the kernel.
>> 
>> Well, as bad as I feared it to be :(
>> 
>> Thanks already, reboot is already announced to the users. Anything I
>> might help you with to corner that bug?
>
> Aggh .. it's been soooo long since I've heard of this happening I can't
> remember.

A sysrq-t would be a good starting point.

-Jeff