Autofs hang, centos 4.4

Autofs hang, centos 4.4

[Erich Weiler]

Hi All-

I'm having this problem with autofs under CentOS 4.4 linux that I was 
hoping someone could shed some light on...  Basically I have a bunch of 
auto.* tables set up in /etc.  They work fine.  What does not work fine 
is that when someone tries to automount a directory that doesn't exist, 
the command hangs forever, and has to be killed.  For instance:

% cd /projects/foo
[hangs]

because /projects/foo is unavailable.  Now let me make it a bit more 
complex:  The machine I'm mounting on is on a private network 
(10.x.x.x), the nfs server is on a public network.  There is no NAT 
routing to the public network.  I was assuming that because the machine 
could not reach the NFS server it would just fail or timeout, not hang 
the terminal...

I know this is an odd setup, but it is the way it is and I'm trying to 
deal with it.  :)  Am I out of luck?  Or can I tweak something to make 
this just fail or timeout?  Any advice would be greatly appreciated!!

Regards,
erich

Re: Autofs hang, centos 4.4

[Jeff Moyer]

==> On Thu, 16 Nov 2006 08:02:48 -0800, Erich Weiler <weiler@soe.ucsc.edu> said:

Erich> Hi All- I'm having this problem with autofs under CentOS 4.4
Erich> linux that I was hoping someone could shed some light on...

The version of autofs would be a bit more useful than the version of
CentOS you are using.

Erich> Basically I have a bunch of auto.* tables set up in /etc.  They
Erich> work fine.  What does not work fine is that when someone tries
Erich> to automount a directory that doesn't exist, the command hangs
Erich> forever, and has to be killed.  For instance:

Erich> % cd /projects/foo [hangs]

Erich> because /projects/foo is unavailable.  Now let me make it a bit
Erich> more complex: The machine I'm mounting on is on a private
Erich> network (10.x.x.x), the nfs server is on a public network.
Erich> There is no NAT routing to the public network.  I was assuming
Erich> that because the machine could not reach the NFS server it
Erich> would just fail or timeout, not hang the terminal...

Erich> I know this is an odd setup, but it is the way it is and I'm
Erich> trying to deal with it.  :) Am I out of luck?  Or can I tweak
Erich> something to make this just fail or timeout?  Any advice would
Erich> be greatly appreciated!!

Add the mount options "hard" and "intr" to your map.  Most likely that
will take care of the problem by allowing you to hit Ctrl-C.  If it
doesn't, then please consult http://people.redhat.com/jmoyer for a
list of things we need to diagnose problems.

Thanks,

Jeff

Re: Autofs hang, centos 4.4

[Erich Weiler]

> The version of autofs would be a bit more useful than the version of
> CentOS you are using.

You're absolutely right, sorry about that.  I'm running autofs 4.1.3.

> Add the mount options "hard" and "intr" to your map.  Most likely that
> will take care of the problem by allowing you to hit Ctrl-C.  If it
> doe%sn't, then please consult http://people.redhat.com/jmoyer for a
> list of things we need to diagnose problems.

'hard' and 'intr' are already in the options list, but unfortunately I 
still can't Ctrl-C to stop it.  Here's a list of info for the bug report:

% rpm -q autofs
autofs-4.1.3-187

% uname -r
2.6.9-42.0.3.ELsmp

My autofs maps are served by our OpenLDAP server.  The map that has the 
problem (because sometimes it tries to reference NFS servers outside our 
private network) is /projects.  From the auto.master entry in LDAP:

nisMapName: auto.master
nisMapEntry: ldap:nisMapName=auto.projects,dc=kilokluster,dc=ucsc,dc=edu 
-browse,hard,intr,rw

And an example of a map that hangs:

nisMapName: auto.projects
nisMapEntry: myserver.ucsc.edu:/export/projects/compbio

Other maps (/home, etc) work fine, probably because those NFS servers 
are reachable.  If I try to do a straight mount of this:

% /bin/mount -t nfs -s -o hard,intr,rw 
myserver.ucsc.edu:/export/projects/compbio /projects/compbio

it hangs, but I can Ctrl-C out of it.  So it seems to problem lies with 
the automounter because I can't Ctrl-C out of an 'automount' to this 
mount, I have to kill it from another terminal (as root).

% cat /etc/nsswitch.conf
passwd:     files ldap
shadow:     files ldap
group:      files ldap

hosts:      files dns

bootparams: files
ethers:     files
netmasks:   files
networks:   files
protocols:  files ldap
rpc:        files
services:   files ldap
netgroup:   files ldap
publickey:  files
automount:  files ldap
aliases:    files

% cat /etc/sysconfig/autofs

LOCALOPTIONS=""
DAEMONOPTIONS="--timeout=60"
LDAPAUTOMASTER=""
UNDERSCORETODOT=1
DISABLE_DIRECT=1
ONE_AUTO_MASTER=0
GHOSTDIRS=""

Again, an explanation of my problem from my previous email:

Basically I have a bunch of auto.* tables set up in /etc.  They work 
fine.  What does not work fine is that when someone (a regular user, not 
root) tries to automount a directory that doesn't exist, the command 
hangs forever, and has to be killed by root.  For instance:

% cd /projects/foo
[hangs]

because /projects/foo is unavailable.  Now let me make it a bit more 
complex:  The machine I'm mounting on is on a private network 
(10.x.x.x), the nfs server is on a public network.  There is no NAT 
routing to the public network.  I was assuming that because the machine 
could not reach the NFS server it would just fail or timeout, not hang 
the terminal...

Odd thing is that if I try to go to this automount as root, it hangs, 
but I *can* Ctrl-C out of it (which is good).  If I try to cd to this 
directory as a regular user, I cannot Ctrl-C out of it.  Which is bad, 
because it's more important for my users to be able to kill it, not 
root.  This is the crux of my problem.  Failing to automount is fine, 
but we need to be able to Ctrl-C out of it as a regular user.

I get this in the /var/log/debug log (after turning on debugging in 
autofs and syslog):

Nov 16 10:47:00 kkldap3 automount[2959]: parse(sun): 
dequote("myserver.ucsc.edu:/export/projects/compbio") -> 
myserver.ucsc.edu:/export/projects/compbio
Nov 16 10:47:00 kkldap3 automount[2959]: parse(sun): core of entry: 
options=hard,intr,rw, loc=myserver.ucsc.edu:/export/projects/compbio
Nov 16 10:47:00 kkldap3 automount[2959]: parse(sun): mounting root 
/projects, mountpoint compbio, what 
myserver.ucsc.edu:/export/projects/compbio, fstype nfs, options hard,intr,rw
Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs):  root=/projects 
name=compbio what=myserver.ucsc.edu:/export/projects/compbio, 
fstype=nfs, options=hard,intr,rw
Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): nfs 
options="hard,intr,rw", nosymlink=0
Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): is_bad_host: 
myserver.ucsc.edu:/export/projects/compbioNov 16 10:47:00 kkldap3 
automount[2959]: mount(nfs): is_local_mount: 
myserver.ucsc.edu:/export/projects/compbio
Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): from 
myserver.ucsc.edu:/export/projects/compbio elected 
myserver.ucsc.edu:/export/projects/compbio
Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): calling mkdir_path 
/projects/compbio
Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): calling mount -t 
nfs -s  -o hard,intr,rw myserver.ucsc.edu:/export/projects/compbio 
/projects/compbio

 From /var/log/messages:

Nov 16 10:53:18 kkldap3 automount[2959]: >> mount: mount to NFS server 
'myserver.ucsc.edu' failed: System Error: Connection timed out.
Nov 16 10:53:18 kkldap3 automount[2959]: mount(nfs): nfs: mount failure 
myserver.ucsc.edu:/export/projects/compbio on /projects/compbio
Nov 16 10:53:18 kkldap3 automount[2959]: failed to mount /projects/compbio

If you need anything else from me please let me know!

Regards,
erich

Re: Autofs hang, centos 4.4

[Ian Kent]

On Thu, 2006-11-16 at 11:03 -0800, Erich Weiler wrote:
> > The version of autofs would be a bit more useful than the version of
> > CentOS you are using.
> 
> You're absolutely right, sorry about that.  I'm running autofs 4.1.3.
> 
> > Add the mount options "hard" and "intr" to your map.  Most likely that
> > will take care of the problem by allowing you to hit Ctrl-C.  If it
> > doe%sn't, then please consult http://people.redhat.com/jmoyer for a
> > list of things we need to diagnose problems.
> 
> 'hard' and 'intr' are already in the options list, but unfortunately I 
> still can't Ctrl-C to stop it.  Here's a list of info for the bug report:
> 
> % rpm -q autofs
> autofs-4.1.3-187
> 
> % uname -r
> 2.6.9-42.0.3.ELsmp
> 
> My autofs maps are served by our OpenLDAP server.  The map that has the 
> problem (because sometimes it tries to reference NFS servers outside our 
> private network) is /projects.  From the auto.master entry in LDAP:
> 
> nisMapName: auto.master
> nisMapEntry: ldap:nisMapName=auto.projects,dc=kilokluster,dc=ucsc,dc=edu 
> -browse,hard,intr,rw
> 
> And an example of a map that hangs:
> 
> nisMapName: auto.projects
> nisMapEntry: myserver.ucsc.edu:/export/projects/compbio
> 
> Other maps (/home, etc) work fine, probably because those NFS servers 
> are reachable.  If I try to do a straight mount of this:
> 
> % /bin/mount -t nfs -s -o hard,intr,rw 
> myserver.ucsc.edu:/export/projects/compbio /projects/compbio
> 
> it hangs, but I can Ctrl-C out of it.  So it seems to problem lies with 
> the automounter because I can't Ctrl-C out of an 'automount' to this 
> mount, I have to kill it from another terminal (as root).
> 
> % cat /etc/nsswitch.conf
> passwd:     files ldap
> shadow:     files ldap
> group:      files ldap
> 
> hosts:      files dns
> 
> bootparams: files
> ethers:     files
> netmasks:   files
> networks:   files
> protocols:  files ldap
> rpc:        files
> services:   files ldap
> netgroup:   files ldap
> publickey:  files
> automount:  files ldap
> aliases:    files
> 
> % cat /etc/sysconfig/autofs
> 
> LOCALOPTIONS=""
> DAEMONOPTIONS="--timeout=60"
> LDAPAUTOMASTER=""
> UNDERSCORETODOT=1
> DISABLE_DIRECT=1
> ONE_AUTO_MASTER=0
> GHOSTDIRS=""
> 
> Again, an explanation of my problem from my previous email:
> 
> Basically I have a bunch of auto.* tables set up in /etc.  They work 
> fine.  What does not work fine is that when someone (a regular user, not 
> root) tries to automount a directory that doesn't exist, the command 
> hangs forever, and has to be killed by root.  For instance:
> 
> % cd /projects/foo
> [hangs]
> 
> because /projects/foo is unavailable.  Now let me make it a bit more 
> complex:  The machine I'm mounting on is on a private network 
> (10.x.x.x), the nfs server is on a public network.  There is no NAT 
> routing to the public network.  I was assuming that because the machine 
> could not reach the NFS server it would just fail or timeout, not hang 
> the terminal...
> 
> Odd thing is that if I try to go to this automount as root, it hangs, 
> but I *can* Ctrl-C out of it (which is good).  If I try to cd to this 
> directory as a regular user, I cannot Ctrl-C out of it.  Which is bad, 
> because it's more important for my users to be able to kill it, not 
> root.  This is the crux of my problem.  Failing to automount is fine, 
> but we need to be able to Ctrl-C out of it as a regular user.

And if you Ctrl-C two or more times does it then stop?

> 
> I get this in the /var/log/debug log (after turning on debugging in 
> autofs and syslog):
> 
> Nov 16 10:47:00 kkldap3 automount[2959]: parse(sun): 
> dequote("myserver.ucsc.edu:/export/projects/compbio") -> 
> myserver.ucsc.edu:/export/projects/compbio
> Nov 16 10:47:00 kkldap3 automount[2959]: parse(sun): core of entry: 
> options=hard,intr,rw, loc=myserver.ucsc.edu:/export/projects/compbio
> Nov 16 10:47:00 kkldap3 automount[2959]: parse(sun): mounting root 
> /projects, mountpoint compbio, what 
> myserver.ucsc.edu:/export/projects/compbio, fstype nfs, options hard,intr,rw
> Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs):  root=/projects 
> name=compbio what=myserver.ucsc.edu:/export/projects/compbio, 
> fstype=nfs, options=hard,intr,rw
> Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): nfs 
> options="hard,intr,rw", nosymlink=0
> Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): is_bad_host: 
> myserver.ucsc.edu:/export/projects/compbioNov 16 10:47:00 kkldap3 
> automount[2959]: mount(nfs): is_local_mount: 
> myserver.ucsc.edu:/export/projects/compbio
> Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): from 
> myserver.ucsc.edu:/export/projects/compbio elected 
> myserver.ucsc.edu:/export/projects/compbio
> Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): calling mkdir_path 
> /projects/compbio
> Nov 16 10:47:00 kkldap3 automount[2959]: mount(nfs): calling mount -t 
> nfs -s  -o hard,intr,rw myserver.ucsc.edu:/export/projects/compbio 
> /projects/compbio

So it is mount that's hanging but there isn't a way to cancel the mount
process. Difficult.

Ian

Re: Autofs hang, centos 4.4

[Erich Weiler]

> And if you Ctrl-C two or more times does it then stop?

Yeah, I can try Ctrl-C and Ctrl-D and ESC a hundred times and nothing 
happens (if I'm a regular user).  As root if I try to automount 
something that hangs, I can hit Ctrl-C and it quits.

> So it is mount that's hanging but there isn't a way to cancel the mount
> process. Difficult.

That's basically it.  It's weird.  And it wouldn't be a big deal but 
some of these mounts will often be unavailable due to our network 
topology, so our users will likely scream if they can't Ctrl-C out of a 
hung mount process.

-erich

Re: Autofs hang, centos 4.4

[Ian Kent]

On Fri, 2006-11-17 at 11:53 -0800, Erich Weiler wrote:
> > And if you Ctrl-C two or more times does it then stop?
> 
> Yeah, I can try Ctrl-C and Ctrl-D and ESC a hundred times and nothing 
> happens (if I'm a regular user).  As root if I try to automount 
> something that hangs, I can hit Ctrl-C and it quits.
> 

Obvious but I have to ask.
Does your login script disable ctrl-c and not reinstate it?

Ian

Re: Autofs hang, centos 4.4

[Erich Weiler]

> Obvious but I have to ask.
> Does your login script disable ctrl-c and not reinstate it?

Unfortunately no.  :)  I use Ctrl-C for a multitude of other things, 
having it disabled would make my life tough...

-erich

Re: Autofs hang, centos 4.4

[Ian Kent]

On Fri, 2006-11-17 at 15:41 +0800, Ian Kent wrote:
> On Thu, 2006-11-16 at 11:03 -0800, Erich Weiler wrote:
> > My autofs maps are served by our OpenLDAP server.  The map that has the 
> > problem (because sometimes it tries to reference NFS servers outside our 
> > private network) is /projects.  From the auto.master entry in LDAP:
> > 
> > nisMapName: auto.master
> > nisMapEntry: ldap:nisMapName=auto.projects,dc=kilokluster,dc=ucsc,dc=edu 
> > -browse,hard,intr,rw

Umm.

Browsable map. It may be possible it's not catching Ctrl-C.

I'll check.

And the user shell is?

Ian

Re: Autofs hang, centos 4.4

[Erich Weiler]

>>> nisMapName: auto.master
>>> nisMapEntry: ldap:nisMapName=auto.projects,dc=kilokluster,dc=ucsc,dc=edu 
>>> -browse,hard,intr,rw
> 
> Umm.
> 
> Browsable map. It may be possible it's not catching Ctrl-C.
> 
> I'll check.
> 
> And the user shell is?

tcsh for normal users.  bash shell when root.  If it's an issue of me 
not using browsable maps I can turn that off, it's not crucial.  I'll 
test it.

Thanks for checking!

-erich

Re: Autofs hang, centos 4.4

[Ian Kent]

On Sat, 2006-11-18 at 17:43 -0800, Erich Weiler wrote:
> >>> nisMapName: auto.master
> >>> nisMapEntry: ldap:nisMapName=auto.projects,dc=kilokluster,dc=ucsc,dc=edu 
> >>> -browse,hard,intr,rw
> > 
> > Umm.
> > 
> > Browsable map. It may be possible it's not catching Ctrl-C.
> > 
> > I'll check.
> > 
> > And the user shell is?
> 
> tcsh for normal users.  bash shell when root.  If it's an issue of me 
> not using browsable maps I can turn that off, it's not crucial.  I'll 
> test it.
> 
> Thanks for checking!

It would be good to temporarily setup an account the same as root but
which uses tcsh as its shell and check that as well.

Ian

Re: Autofs hang, centos 4.4

[Erich Weiler]

> It would be good to temporarily setup an account the same as root but
> which uses tcsh as its shell and check that as well.

Same thing happens.  Root must have some special power to kill hung 
mounts that regular users don't.  Maybe because the automountd process 
is owned by root?

-erich

Re: Autofs hang, centos 4.4

[Ian Kent]

On Mon, 2006-11-20 at 09:50 -0800, Erich Weiler wrote:
> > It would be good to temporarily setup an account the same as root but
> > which uses tcsh as its shell and check that as well.
> 
> Same thing happens.  Root must have some special power to kill hung 
> mounts that regular users don't.  Maybe because the automountd process 
> is owned by root?

Not that I'm aware of.
Don't think the ownership of automount has anything to do with this
issue.

Ian