Re: automount errors under high load

[Ian Kent <raven@themaw.net>]

On Fri, 2006-11-03 at 18:55 -0600, Fletcher Mattox wrote:
> Jeff Moyer writes:
> > ==> Regarding [autofs] automount errors under high load; "Fletcher Mattox" <fletcher+autofs@cs.utexas.edu> adds:
> > 
> > fletcher> Hi,
> > fletcher> Under high load (100 mount/sec) automount will fail with either
> > fletcher> of these error messages from mount.
> > 
> > fletcher>     mount: filer3:/vol/vol2/v2q017: can't read superblock
> > fletcher>     mount: RPC: Authentication error; why = Client credential too weak
> > 
> > fletcher> I find these error messages suspicious, and do not believe either of
> > fletcher> them represent the true problem.  I have appended the debug output in
> > fletcher> case that is useful.  I notice that it seems to try the mount seven
> > fletcher> times each, before giving up with one of the above errors (usually
> > fletcher> it is the superblock error).
> > 
> > The mount program performs the retries.  What errors are reported in the
> > server logs?  This really feels like more of an NFS problem than an autofs
> > one.
> 
> Jeff,
> 
> This is the only error in the server logs:
> 
>   Mon Oct 30 10:58:05 CST [mountd_main:warning]: Client 128.83.120.210 (xid 160302913),\
>   from mount is trying to mount from a nonreserved port = 43645 as uid = 0
> 
> which would explain the weak credentials error.  I wonder why mount would
> use a non-privileged port?  If it runs out of privileged ports, does it
> start using non-privileged ports?  Seems like it should just return an error.

mount(8) is notorious for it's port usage when probing an NFS server.

It can use up to 6 ports per mount and combined with autofs checking if
the server is up this can increase to as many as eight per mount.

Many distributions have a mount(8) that will attempt to use
non-privileged ports for this reason which is bad news if your NFS
server is configured to not allow then.

> 
> In any event, Im now wondering if the problem is caused because all 1024 privileged
> ports are in use.  Hmm.
> 
> But the superblock error is still unexplained.

It's quite likely that mount is reporting this because it can't get NFS
file handle for the mount root. Possibly an error code returned by the
remote mountd.

> 
> > fletcher> I have also noticed corruption of /etc/mtab.  Sometimes static
> > fletcher> mounts just disappear from it during busy times.  I assume this
> > fletcher> is some type of race condition where file locking fails.  I do
> > fletcher> not know if this is the same problem or a different problem, but
> > fletcher> I thought I would mention it.
> > 
> > I posted one method for fixing that problem to the list some time ago.  The
> > idea is to simply use /proc/mounts instead aof /etc/mtab.  That way we
> > avoid the whole lock-file mess.  I've attached the patch, here, but I
> > didn't try to apply it to 4.1.4, so it may need some massaging.  (I'll post
> > a proper patch later on, but I have to work through my backlog first.)
> 
> Thanks.  I just joined the list.  And I have to get up to speed using
> ubuntu's source building environment before I can start applying patches.
> But it's good to know they exist.
> 
> > I don't think the corruption of /etc/mtab will cause the problems you are
> > seeing, but I could be wrong.
> 
> I agree.   I just though it was worth mentioning.
> 
> Thank you for responding!
> Fletcher
> 
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs