debug failed automount

debug failed automount

[Robert Swan]

I'm trying to debug automount refusal to mount volumes.  The manual
mount works fine:

> ypcat -k auto.home | grep robert
robert remotehost:/export/users/car073/robert

then

> mount remotehost:/export/users/car073/robert /home/robert

and ls /home/robert works as expected. However, if I

> automount /var/autofs/home/ yp auto.home

and then try to ls /var/autofs/home/robert, I get 

ls:/var/autofs/home/robert: No such file or directory, 

and /var/log/syslog contains

localhost automount[23333]: failed to mount /var/autofs/home/robert

I've also verified that I'm able to automount locally exported
directories (using a "file" map, however).

One point is that the remote host is accessed through a VPN tunnel.

I've attached automount syslog entries using the -d and -v automount
command line options. I suspect "winner=(null)" is a clue but I have
no idea what it means.

Thanks for any help,
Robert

Apr 28 07:52:28 localhost automount[12608]: handle_packet: type = 0
Apr 28 07:52:28 localhost automount[12608]: handle_packet_missing:
token 4, name robert
Apr 28 07:52:28 localhost automount[12608]: attempting to mount entry
/var/autofs/home/robert
Apr 28 07:52:28 localhost automount[12617]: lookup(yp): looking up robert
Apr 28 07:52:28 localhost automount[12617]: ret = 1
Apr 28 07:52:28 localhost automount[12617]: lookup(yp): robert ->
remotehost:/export/users/car073/robert
Apr 28 07:52:28 localhost automount[12617]: parse(sun): expanded
entry: remotehost:/export/users/car073/robert
Apr 28 07:52:28 localhost automount[12617]: parse(sun): gathered options:
Apr 28 07:52:28 localhost automount[12617]: parse(sun):
dequote("remotehost:/export/users/car073/robert") ->
remotehost:/export/users/car073/robert
Apr 28 07:52:28 localhost automount[12617]: parse(sun): core of entry:
options=, loc=remotehost:/export/users/car073/robert
Apr 28 07:52:28 localhost automount[12617]: parse(sun): mounting root
/var/autofs/home, mountpoint robert, what
remotehost:/export/users/car073/robert, fstype nfs, options (null)
Apr 28 07:52:28 localhost automount[12617]: mount(nfs):
root=/var/autofs/home name=robert
what=remotehost:/export/users/car073/robert, fstype=nfs,
options=(null)
Apr 28 07:52:29 localhost automount[12617]: mount(nfs): winner =
(null) local = 0
Apr 28 07:52:29 localhost automount[12617]: mount(nfs): no host elected
Apr 28 07:52:29 localhost automount[12617]: failed to mount
/var/autofs/home/robert
Apr 28 07:52:29 localhost automount[12617]: umount_multi:
path=/var/autofs/home/robert incl=1
Apr 28 07:52:29 localhost automount[12617]: umount_multi: no mounts
found under /var/autofs/home/robert
Apr 28 07:52:29 localhost automount[12608]: handle_child: got pid
12617, sig 0 (0), stat 1
Apr 28 07:52:29 localhost automount[12608]: sig_child: found pending
iop pid 12617: signalled 0 (sig 0), exit status 1
Apr 28 07:52:29 localhost automount[12608]: send_fail: token=4

Re: debug failed automount

[Robert Swan]

> 
> One point is that the remote host is accessed through a VPN tunnel.

so as it turns out, this is the problem. The rpc_time call fails (I
assume this is because it is not supported by the VPN) and this marks
my only host as dead and automount does not attempt to mount. Even
though the mount actually works fine.

I made myself a little patch to overcome the problem. I think someone
suggested that the rpc_ping stuff should be used to to rank the hosts,
and that mount should try all of them before giving up. Seems like a
good idea.

Robert

Re: Re: debug failed automount

[raven]

On Sat, 30 Apr 2005, Robert Swan wrote:

>>
>> One point is that the remote host is accessed through a VPN tunnel.
>
> so as it turns out, this is the problem. The rpc_time call fails (I
> assume this is because it is not supported by the VPN) and this marks
> my only host as dead and automount does not attempt to mount. Even
> though the mount actually works fine.

Why? The rpc_ping connects to port 2049 on the target host.

>
> I made myself a little patch to overcome the problem. I think someone
> suggested that the rpc_ping stuff should be used to to rank the hosts,
> and that mount should try all of them before giving up. Seems like a
> good idea.

Mount will only try the hosts that autofs gives it.

We are going full circle. Before there were complaints about mounting 
taking to long so a heap of work was done on that. Just when that's 
almost working OK it looks like people are asking for the opposite.

If autofs can't connect to the RPC null procedure on the NFS port of 
the target host autofs won't ask mount to mount it.

Ian

Re: Re: debug failed automount

[raven]

On Sat, 30 Apr 2005 raven@themaw.net wrote:

My response was a bit hasty I think.

> On Sat, 30 Apr 2005, Robert Swan wrote:
>
>> > 
>> >  One point is that the remote host is accessed through a VPN tunnel.
>> 
>>  so as it turns out, this is the problem. The rpc_time call fails (I
>>  assume this is because it is not supported by the VPN) and this marks
>>  my only host as dead and automount does not attempt to mount. Even
>>  though the mount actually works fine.
>
> Why? The rpc_ping connects to port 2049 on the target host.

You may be seeing a bug with the autofs mount code not trying a longer 
timeout if it fails to get a response in time.

The latest RedHat rpms may have the fix? Jeff, Chris?

There's a patch on kernel.org for it as well.

>
>> 
>>  I made myself a little patch to overcome the problem. I think someone
>>  suggested that the rpc_ping stuff should be used to to rank the hosts,
>>  and that mount should try all of them before giving up. Seems like a
>>  good idea.
>
> Mount will only try the hosts that autofs gives it.
>
> We are going full circle. Before there were complaints about mounting taking 
> to long so a heap of work was done on that. Just when that's almost working 
> OK it looks like people are asking for the opposite.
>
> If autofs can't connect to the RPC null procedure on the NFS port of the 
> target host autofs won't ask mount to mount it.
>
> Ian
>
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs
>

Re: Re: debug failed automount

[Robert Swan]

On 5/1/05, raven@themaw.net <raven@themaw.net> wrote:
> On Sat, 30 Apr 2005 raven@themaw.net wrote:
> 
> My response was a bit hasty I think.
> 
> > On Sat, 30 Apr 2005, Robert Swan wrote:
> >
> >> >
> >> >  One point is that the remote host is accessed through a VPN tunnel.
> >>
> >>  so as it turns out, this is the problem. The rpc_time call fails (I
> >>  assume this is because it is not supported by the VPN) and this marks
> >>  my only host as dead and automount does not attempt to mount. Even
> >>  though the mount actually works fine.
> >
> > Why? The rpc_ping connects to port 2049 on the target host.
> 
> You may be seeing a bug with the autofs mount code not trying a longer
> timeout if it fails to get a response in time.

I'm a bit confused myself because I can no longer replicate my
problem. I had made a test program to call your get_best_mount() with
my particular arguments so I could step through it, and  I'm _sure_
(aren't we all) that it would get a failed status from rpc_time at
line 238 and so would not assign a winner. I changed initial
assignment of "winner" at the start of the function which solved my
problem. I didn't look inside rpc_time to see what was going wrong; I
tried to do that today and now the status from rpc_time() is
successful, and your original code performs the mounts. So I don't
have a bug report and you can ignore me.

Is there a way to attach a debugger to the automount process that gets
spawned when the kernel attempts an automount?

Thanks anyhow for answering.

Robert

> 

> > _______________________________________________
> > autofs mailing list
> > autofs@linux.kernel.org
> > http://linux.kernel.org/mailman/listinfo/autofs
> >
> 
>

Re: Re: debug failed automount

[Ian Kent]

On Sun, 1 May 2005, Robert Swan wrote:

> On 5/1/05, raven@themaw.net <raven@themaw.net> wrote:
> > On Sat, 30 Apr 2005 raven@themaw.net wrote:
> > 
> > My response was a bit hasty I think.
> > 
> > > On Sat, 30 Apr 2005, Robert Swan wrote:
> > >
> > >> >
> > >> >  One point is that the remote host is accessed through a VPN tunnel.
> > >>
> > >>  so as it turns out, this is the problem. The rpc_time call fails (I
> > >>  assume this is because it is not supported by the VPN) and this marks
> > >>  my only host as dead and automount does not attempt to mount. Even
> > >>  though the mount actually works fine.
> > >
> > > Why? The rpc_ping connects to port 2049 on the target host.
> > 
> > You may be seeing a bug with the autofs mount code not trying a longer
> > timeout if it fails to get a response in time.
> 
> I'm a bit confused myself because I can no longer replicate my
> problem. I had made a test program to call your get_best_mount() with
> my particular arguments so I could step through it, and  I'm _sure_
> (aren't we all) that it would get a failed status from rpc_time at
> line 238 and so would not assign a winner. I changed initial
> assignment of "winner" at the start of the function which solved my
> problem. I didn't look inside rpc_time to see what was going wrong; I
> tried to do that today and now the status from rpc_time() is
> successful, and your original code performs the mounts. So I don't
> have a bug report and you can ignore me.

This sound even more like the original problem.

The symptom was intermitent fails.

So if the target server responds quickly enough all is well.
But if there's a bit to much trafic or if you stop to scratch it doesn't 
work.

As I said I'd recommend the two patches on kernel.org.

> 
> Is there a way to attach a debugger to the automount process that gets
> spawned when the kernel attempts an automount?

Nothing that will help in this situation.
All the mount/umount is done in userspace by the daemon.

Ian

Re: Re: debug failed automount

[Robert Swan]

On 5/1/05, Ian Kent <raven@themaw.net> wrote:
> On Sun, 1 May 2005, Robert Swan wrote:
> 

> > Is there a way to attach a debugger to the automount process that gets
> > spawned when the kernel attempts an automount?
> 
> Nothing that will help in this situation.
> All the mount/umount is done in userspace by the daemon.

I couldn't figure out how to attach to the user space code either.
I.E. I wanted to step through get_best_mount() but wasn't able; still
would like to know how to do that.

Robert

> 
> Ian
> 
>

Re: Re: debug failed automount

[raven]

On Sun, 1 May 2005, Robert Swan wrote:

> On 5/1/05, Ian Kent <raven@themaw.net> wrote:
>> On Sun, 1 May 2005, Robert Swan wrote:
>>
>
>>> Is there a way to attach a debugger to the automount process that gets
>>> spawned when the kernel attempts an automount?
>>
>> Nothing that will help in this situation.
>> All the mount/umount is done in userspace by the daemon.
>
> I couldn't figure out how to attach to the user space code either.
> I.E. I wanted to step through get_best_mount() but wasn't able; still
> would like to know how to do that.

That's a bit hard.
There are a couple of reasons. First lots of stuff is done in 
sub-proccesses, second the exe is usually a position independent exe.

I haven't got around to fixing the debug compile options to see if I can 
remedy that.

But then I do all my debuging with debug prints.

Have you tried the patch?

Ian

Re: Re: debug failed automount

[Steven Ihde]

On Mon, 02 May 2005 10:03:38 +0800, Ian Kent wrote:
[discussion of failure to automount a NFS server over a VPN tunnel...]
> This sound even more like the original problem.
> 
> The symptom was intermitent fails.
> 
> So if the target server responds quickly enough all is well.
> But if there's a bit to much trafic or if you stop to scratch it doesn't 
> work.
> 
> As I said I'd recommend the two patches on kernel.org.

I was suffering the same problem, and the patch fixed it.

I'm automounting an NFS server in India with a 250ms round trip time --
autofs consistently refused to mount it.

I applied the non-replicated-ping patch from kernel.org and the problem is
fixed:

http://www.kernel.org/pub/linux/daemons/autofs/v4/autofs-4.1.4-non-replicated-ping.patch

Thanks for the information.

Along the way I noticed that the replicated server support seems a bit
broken with respect to the "nosymlink" option.  Before applying the patch
above I noticed that "nosymlink" allowed the slow server to mount -- this
is because "nosymlink" completely bypasses get_best_mount.  This is OK if
you have only a single server, but doesn't work if you have a replicated
mount.  For example,

* -nosymlink foo.bar.com,baz.quux.com:/some/path

will wind up passing the literal string
'foo.bar.com,baz.quux.com:/some/path' to mount, which of course fails --
if you remove the nosymlink option it works fine.

Thanks,

Steve