automount errors under high load

automount errors under high load

[Fletcher Mattox]

Hi,

Under high load (100 mount/sec) automount will fail with either of these
error messages from mount.

    mount: filer3:/vol/vol2/v2q017: can't read superblock
    mount: RPC: Authentication error; why = Client credential too weak

I find these error messages suspicious, and do not believe either of
them represent the true problem.  I have appended the debug output in
case that is useful.  I notice that it seems to try the mount seven
times each, before giving up with one of the above errors (usually
it is the superblock error).

I have also noticed corruption of /etc/mtab.  Sometimes static mounts
just disappear from it during busy times.  I assume this is some type
of race condition where file locking fails.  I do not know if this is
the same problem or a different problem, but I thought I would mention it.

I do not know what causes such a high load of mount requests.  But I see
them in the accounting files.  Nothing obvious is going on when the errors
occur.  They happen for several seconds at a time, 6-8 times per day.

The NFS client is ubuntu 6.06 (dapper), kernel 2.6.17.4, automount 4.1.4.
The NFS server is a Network Appliances filer.

Thanks,
Fletcher

Nov  2 08:41:09 smtp automount[3862]: handle_child: got pid 15301, sig 0 (0), stat 0 
Nov  2 08:41:09 smtp automount[3862]: sig_child: found pending iop pid 15301: signalled 0 (sig 0), exit status 0
Nov  2 08:41:09 smtp automount[3862]: send_ready: token=4324 
Nov  2 08:41:11 smtp automount[3862]: handle_packet: type = 0 
Nov  2 08:41:11 smtp automount[3862]: handle_packet_missing: token 4325, name v2q007 
Nov  2 08:41:11 smtp automount[3862]: attempting to mount entry /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: lookup(yp): looking up v2q007
Nov  2 08:41:11 smtp automount[15326]: ret = 1
Nov  2 08:41:11 smtp automount[15326]: lookup(yp): v2q007 -> -intr,grpid,quota,retry=2^Ifiler3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: parse(sun): expanded entry: -intr,grpid,quota,retry=2^Ifiler3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: parse(sun): dequote("intr,grpid,quota,retry=2") -> intr,grpid,quota,retry=2
Nov  2 08:41:11 smtp automount[15326]: parse(sun): gathered options: tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2
Nov  2 08:41:11 smtp automount[15326]: parse(sun): dequote("filer3:/vol/vol2/v2q007") -> filer3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: parse(sun): core of entry: options=tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2, loc=filer3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: parse(sun): mounting root /v/filer3, mountpoint v2q007, what filer3:/vol/vol2/v2q007, fstype nfs, options tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2 
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): root=/v/filer3 name=v2q007 what=filer3:/vol/vol2/v2q007, fstype=nfs, options=tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): nfs options="tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2", nosymlink=0, symlink=0, ro=0
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): is_local_mount: filer3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): from filer3:/vol/vol2/v2q007 elected filer3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): calling mkdir_path /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): calling mount -t nfs -s  -o tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2 filer3:/vol/vol2/v2q007 /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: >> mount: filer3:/vol/vol2/v2q007: can't read superblock
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): nfs: mount failure filer3:/vol/vol2/v2q007 on /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: failed to mount /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: umount_multi: path=/v/filer3/v2q007 incl=1 
Nov  2 08:41:11 smtp automount[15326]: umount_multi: no mounts found under /v/filer3/v2q007

Re: automount errors under high load

[Jeff Moyer]

==> Regarding [autofs] automount errors under high load; "Fletcher Mattox" <fletcher+autofs@cs.utexas.edu> adds:

fletcher> Hi,
fletcher> Under high load (100 mount/sec) automount will fail with either
fletcher> of these error messages from mount.

fletcher>     mount: filer3:/vol/vol2/v2q017: can't read superblock
fletcher>     mount: RPC: Authentication error; why = Client credential too weak

fletcher> I find these error messages suspicious, and do not believe either of
fletcher> them represent the true problem.  I have appended the debug output in
fletcher> case that is useful.  I notice that it seems to try the mount seven
fletcher> times each, before giving up with one of the above errors (usually
fletcher> it is the superblock error).

The mount program performs the retries.  What errors are reported in the
server logs?  This really feels like more of an NFS problem than an autofs
one.

fletcher> I have also noticed corruption of /etc/mtab.  Sometimes static
fletcher> mounts just disappear from it during busy times.  I assume this
fletcher> is some type of race condition where file locking fails.  I do
fletcher> not know if this is the same problem or a different problem, but
fletcher> I thought I would mention it.

I posted one method for fixing that problem to the list some time ago.  The
idea is to simply use /proc/mounts instead aof /etc/mtab.  That way we
avoid the whole lock-file mess.  I've attached the patch, here, but I
didn't try to apply it to 4.1.4, so it may need some massaging.  (I'll post
a proper patch later on, but I have to work through my backlog first.)

I don't think the corruption of /etc/mtab will cause the problems you are
seeing, but I could be wrong.

fletcher> I do not know what causes such a high load of mount requests.
fletcher> But I see them in the accounting files.  Nothing obvious is going
fletcher> on when the errors occur.  They happen for several seconds at a
fletcher> time, 6-8 times per day.

fletcher> The NFS client is ubuntu 6.06 (dapper), kernel 2.6.17.4,
fletcher> automount 4.1.4.  The NFS server is a Network Appliances filer.

-Jeff

--- autofs-4.1.3/daemon/automount.c.orig	2006-05-22 10:55:31.000000000 -0400
+++ autofs-4.1.3/daemon/automount.c	2006-05-22 10:55:43.000000000 -0400
@@ -379,7 +379,7 @@ static int umount_multi(const char *path
 	debug("umount_multi: path=%s incl=%d\n", path, incl);
 
 	wait_for_lock();
-	mtab = setmntent(_PATH_MOUNTED, "r");
+	mtab = setmntent(path_mounted, "r");
 	if (!mtab) {
 		unlink(AUTOFS_LOCK);
 		error("umount_multi: setmntent: %m");
@@ -1886,6 +1886,16 @@ int handle_mounts(char *path)
 	return 0;
 }
 
+void init_path_mounted(void)
+{
+	struct stat st;
+
+	if (stat(PROC_MOUNTS, &st) == 0)
+		path_mounted = PROC_MOUNTS;
+	else
+		path_mounted = _PATH_MOUNTED;
+}
+
 int main(int argc, char *argv[])
 {
 	char *path, *map, *mapfmt;
@@ -1965,6 +1975,8 @@ int main(int argc, char *argv[])
 
 	become_daemon();
 
+	init_path_mounted();
+
 	path = argv[0];
 	map = argv[1];
 	mapargv = (const char **) &argv[2];
--- autofs-4.1.3/include/automount.h.orig	2006-05-22 10:55:31.000000000 -0400
+++ autofs-4.1.3/include/automount.h	2006-05-22 10:55:43.000000000 -0400
@@ -38,6 +38,7 @@
 #endif
 
 #define DEFAULT_TIMEOUT (5*60)			/* 5 minutes */
+#define PROC_MOUNTS	"/proc/mounts"
 #define AUTOFS_LOCK	"/var/lock/autofs"	/* To serialize access to mount */
 #define MOUNTED_LOCK	_PATH_MOUNTED "~"	/* mounts' lock file */
 #define MTAB_NOTUPDATED 0x1000			/* mtab succeded but not updated */
@@ -119,6 +120,7 @@ struct autofs_point {
 };
 
 extern struct autofs_point ap; 
+const char *path_mounted;
 
 /* Standard function used by daemon or modules */
 
--- autofs-4.1.3/lib/cache.c.orig	2006-05-22 10:55:54.000000000 -0400
+++ autofs-4.1.3/lib/cache.c	2006-05-22 10:56:01.000000000 -0400
@@ -475,7 +475,7 @@ int is_mounted(const char *path)
 		return ret;
 
 	wait_for_lock();
-	mtab = setmntent(_PATH_MOUNTED, "r");
+	mtab = setmntent(path_mounted, "r");
 	if (!mtab) {
 		unlink(AUTOFS_LOCK);
 		error("is_mounted: setmntent: %m");

Re: automount errors under high load

[Fletcher Mattox]

Jeff Moyer writes:
> ==> Regarding [autofs] automount errors under high load; "Fletcher Mattox" <fletcher+autofs@cs.utexas.edu> adds:
> 
> fletcher> Hi,
> fletcher> Under high load (100 mount/sec) automount will fail with either
> fletcher> of these error messages from mount.
> 
> fletcher>     mount: filer3:/vol/vol2/v2q017: can't read superblock
> fletcher>     mount: RPC: Authentication error; why = Client credential too weak
> 
> fletcher> I find these error messages suspicious, and do not believe either of
> fletcher> them represent the true problem.  I have appended the debug output in
> fletcher> case that is useful.  I notice that it seems to try the mount seven
> fletcher> times each, before giving up with one of the above errors (usually
> fletcher> it is the superblock error).
> 
> The mount program performs the retries.  What errors are reported in the
> server logs?  This really feels like more of an NFS problem than an autofs
> one.

Jeff,

This is the only error in the server logs:

  Mon Oct 30 10:58:05 CST [mountd_main:warning]: Client 128.83.120.210 (xid 160302913),\
  from mount is trying to mount from a nonreserved port = 43645 as uid = 0

which would explain the weak credentials error.  I wonder why mount would
use a non-privileged port?  If it runs out of privileged ports, does it
start using non-privileged ports?  Seems like it should just return an error.

In any event, Im now wondering if the problem is caused because all 1024 privileged
ports are in use.  Hmm.

But the superblock error is still unexplained.

> fletcher> I have also noticed corruption of /etc/mtab.  Sometimes static
> fletcher> mounts just disappear from it during busy times.  I assume this
> fletcher> is some type of race condition where file locking fails.  I do
> fletcher> not know if this is the same problem or a different problem, but
> fletcher> I thought I would mention it.
> 
> I posted one method for fixing that problem to the list some time ago.  The
> idea is to simply use /proc/mounts instead aof /etc/mtab.  That way we
> avoid the whole lock-file mess.  I've attached the patch, here, but I
> didn't try to apply it to 4.1.4, so it may need some massaging.  (I'll post
> a proper patch later on, but I have to work through my backlog first.)

Thanks.  I just joined the list.  And I have to get up to speed using
ubuntu's source building environment before I can start applying patches.
But it's good to know they exist.

> I don't think the corruption of /etc/mtab will cause the problems you are
> seeing, but I could be wrong.

I agree.   I just though it was worth mentioning.

Thank you for responding!
Fletcher

Re: automount errors under high load

[Ian Kent]

On Fri, 2006-11-03 at 18:55 -0600, Fletcher Mattox wrote:
> Jeff Moyer writes:
> > ==> Regarding [autofs] automount errors under high load; "Fletcher Mattox" <fletcher+autofs@cs.utexas.edu> adds:
> > 
> > fletcher> Hi,
> > fletcher> Under high load (100 mount/sec) automount will fail with either
> > fletcher> of these error messages from mount.
> > 
> > fletcher>     mount: filer3:/vol/vol2/v2q017: can't read superblock
> > fletcher>     mount: RPC: Authentication error; why = Client credential too weak
> > 
> > fletcher> I find these error messages suspicious, and do not believe either of
> > fletcher> them represent the true problem.  I have appended the debug output in
> > fletcher> case that is useful.  I notice that it seems to try the mount seven
> > fletcher> times each, before giving up with one of the above errors (usually
> > fletcher> it is the superblock error).
> > 
> > The mount program performs the retries.  What errors are reported in the
> > server logs?  This really feels like more of an NFS problem than an autofs
> > one.
> 
> Jeff,
> 
> This is the only error in the server logs:
> 
>   Mon Oct 30 10:58:05 CST [mountd_main:warning]: Client 128.83.120.210 (xid 160302913),\
>   from mount is trying to mount from a nonreserved port = 43645 as uid = 0
> 
> which would explain the weak credentials error.  I wonder why mount would
> use a non-privileged port?  If it runs out of privileged ports, does it
> start using non-privileged ports?  Seems like it should just return an error.

mount(8) is notorious for it's port usage when probing an NFS server.

It can use up to 6 ports per mount and combined with autofs checking if
the server is up this can increase to as many as eight per mount.

Many distributions have a mount(8) that will attempt to use
non-privileged ports for this reason which is bad news if your NFS
server is configured to not allow then.

> 
> In any event, Im now wondering if the problem is caused because all 1024 privileged
> ports are in use.  Hmm.
> 
> But the superblock error is still unexplained.

It's quite likely that mount is reporting this because it can't get NFS
file handle for the mount root. Possibly an error code returned by the
remote mountd.

> 
> > fletcher> I have also noticed corruption of /etc/mtab.  Sometimes static
> > fletcher> mounts just disappear from it during busy times.  I assume this
> > fletcher> is some type of race condition where file locking fails.  I do
> > fletcher> not know if this is the same problem or a different problem, but
> > fletcher> I thought I would mention it.
> > 
> > I posted one method for fixing that problem to the list some time ago.  The
> > idea is to simply use /proc/mounts instead aof /etc/mtab.  That way we
> > avoid the whole lock-file mess.  I've attached the patch, here, but I
> > didn't try to apply it to 4.1.4, so it may need some massaging.  (I'll post
> > a proper patch later on, but I have to work through my backlog first.)
> 
> Thanks.  I just joined the list.  And I have to get up to speed using
> ubuntu's source building environment before I can start applying patches.
> But it's good to know they exist.
> 
> > I don't think the corruption of /etc/mtab will cause the problems you are
> > seeing, but I could be wrong.
> 
> I agree.   I just though it was worth mentioning.
> 
> Thank you for responding!
> Fletcher
> 
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs

Re: automount errors under high load

[Fletcher Mattox]

Jeff Moyer writes:
> ==> Regarding [autofs] automount errors under high load; "Fletcher Mattox" <fletcher+autofs@cs.utexas.edu> adds:
> 
> fletcher> Hi,
> fletcher> Under high load (100 mount/sec) automount will fail with either
> fletcher> of these error messages from mount.
> 
> fletcher>     mount: filer3:/vol/vol2/v2q017: can't read superblock
> fletcher>     mount: RPC: Authentication error; why = Client credential too weak
> 
> fletcher> I find these error messages suspicious, and do not believe either of
> fletcher> them represent the true problem.  I have appended the debug output in
> fletcher> case that is useful.  I notice that it seems to try the mount seven
> fletcher> times each, before giving up with one of the above errors (usually
> fletcher> it is the superblock error).
> 
> The mount program performs the retries.  What errors are reported in the
> server logs?  This really feels like more of an NFS problem than an autofs
> one.

Jeff,

This is the only error in the server logs:

  Mon Oct 30 10:58:05 CST [mountd_main:warning]: Client 128.83.120.210 (xid 160302913),\
  from mount is trying to mount from a nonreserved port = 43645 as uid = 0

which would explain the weak credentials error.  I wonder why mount would
use a non-privileged port?  If it runs out of privileged ports, does it
start using non-privileged ports?  Seems like it should just return an error.

In any event, Im now wondering if the problem is caused because all 1024 privileged
ports are in use.  Hmm.

But the superblock error is still unexplained.

> fletcher> I have also noticed corruption of /etc/mtab.  Sometimes static
> fletcher> mounts just disappear from it during busy times.  I assume this
> fletcher> is some type of race condition where file locking fails.  I do
> fletcher> not know if this is the same problem or a different problem, but
> fletcher> I thought I would mention it.
> 
> I posted one method for fixing that problem to the list some time ago.  The
> idea is to simply use /proc/mounts instead aof /etc/mtab.  That way we
> avoid the whole lock-file mess.  I've attached the patch, here, but I
> didn't try to apply it to 4.1.4, so it may need some massaging.  (I'll post
> a proper patch later on, but I have to work through my backlog first.)

Thanks.  I just joined the list.  And I have to get up to speed using
ubuntu's source building environment before I can start applying patches.
But it's good to know they exist.

> I don't think the corruption of /etc/mtab will cause the problems you are
> seeing, but I could be wrong.

I agree.   I just though it was worth mentioning.

Thank you for responding!
Fletcher

Re: automount errors under high load

[Ian Kent]

On Fri, 2006-11-03 at 15:22 -0600, Fletcher Mattox wrote:

> I have also noticed corruption of /etc/mtab.  Sometimes static mounts
> just disappear from it during busy times.  I assume this is some type
> of race condition where file locking fails.  I do not know if this is
> the same problem or a different problem, but I thought I would mention it.

If you see this type of corruption then you need to check if mount(8)
locking is broken or not and if the autofs package you are using has the
locking at mount time disabled.

Have a look at
http://www.kernel.org/pub/linux/daemons/autofs/v4/autofs-4.1.4-configureable-locking.patch
for more information.

> 
> I do not know what causes such a high load of mount requests.  But I see
> them in the accounting files.  Nothing obvious is going on when the errors
> occur.  They happen for several seconds at a time, 6-8 times per day.
> 
> The NFS client is ubuntu 6.06 (dapper), kernel 2.6.17.4, automount 4.1.4.
> The NFS server is a Network Appliances filer.
> 
> Thanks,
> Fletcher
> 
> Nov  2 08:41:09 smtp automount[3862]: handle_child: got pid 15301, sig 0 (0), stat 0 
> Nov  2 08:41:09 smtp automount[3862]: sig_child: found pending iop pid 15301: signalled 0 (sig 0), exit status 0
> Nov  2 08:41:09 smtp automount[3862]: send_ready: token=4324 
> Nov  2 08:41:11 smtp automount[3862]: handle_packet: type = 0 
> Nov  2 08:41:11 smtp automount[3862]: handle_packet_missing: token 4325, name v2q007 
> Nov  2 08:41:11 smtp automount[3862]: attempting to mount entry /v/filer3/v2q007
> Nov  2 08:41:11 smtp automount[15326]: lookup(yp): looking up v2q007
> Nov  2 08:41:11 smtp automount[15326]: ret = 1
> Nov  2 08:41:11 smtp automount[15326]: lookup(yp): v2q007 -> -intr,grpid,quota,retry=2^Ifiler3:/vol/vol2/v2q007
> Nov  2 08:41:11 smtp automount[15326]: parse(sun): expanded entry: -intr,grpid,quota,retry=2^Ifiler3:/vol/vol2/v2q007
> Nov  2 08:41:11 smtp automount[15326]: parse(sun): dequote("intr,grpid,quota,retry=2") -> intr,grpid,quota,retry=2
> Nov  2 08:41:11 smtp automount[15326]: parse(sun): gathered options: tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2
> Nov  2 08:41:11 smtp automount[15326]: parse(sun): dequote("filer3:/vol/vol2/v2q007") -> filer3:/vol/vol2/v2q007
> Nov  2 08:41:11 smtp automount[15326]: parse(sun): core of entry: options=tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2, loc=filer3:/vol/vol2/v2q007
> Nov  2 08:41:11 smtp automount[15326]: parse(sun): mounting root /v/filer3, mountpoint v2q007, what filer3:/vol/vol2/v2q007, fstype nfs, options tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2 
> Nov  2 08:41:11 smtp automount[15326]: mount(nfs): root=/v/filer3 name=v2q007 what=filer3:/vol/vol2/v2q007, fstype=nfs, options=tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2
> Nov  2 08:41:11 smtp automount[15326]: mount(nfs): nfs options="tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2", nosymlink=0, symlink=0, ro=0
> Nov  2 08:41:11 smtp automount[15326]: mount(nfs): is_local_mount: filer3:/vol/vol2/v2q007
> Nov  2 08:41:11 smtp automount[15326]: mount(nfs): from filer3:/vol/vol2/v2q007 elected filer3:/vol/vol2/v2q007
> Nov  2 08:41:11 smtp automount[15326]: mount(nfs): calling mkdir_path /v/filer3/v2q007
> Nov  2 08:41:11 smtp automount[15326]: mount(nfs): calling mount -t nfs -s  -o tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2 filer3:/vol/vol2/v2q007 /v/filer3/v2q007
> Nov  2 08:41:11 smtp automount[15326]: >> mount: filer3:/vol/vol2/v2q007: can't read superblock

A message from mount, probably caused by the port exhaustion you
described elsewhere.

> Nov  2 08:41:11 smtp automount[15326]: mount(nfs): nfs: mount failure filer3:/vol/vol2/v2q007 on /v/filer3/v2q007
> Nov  2 08:41:11 smtp automount[15326]: failed to mount /v/filer3/v2q007
> Nov  2 08:41:11 smtp automount[15326]: umount_multi: path=/v/filer3/v2q007 incl=1 
> Nov  2 08:41:11 smtp automount[15326]: umount_multi: no mounts found under /v/filer3/v2q007
> 
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs

automount errors under high load

[Fletcher Mattox]

Hi,

Under high load (100 mount/sec) automount will fail with either of these
error messages from mount.

    mount: filer3:/vol/vol2/v2q017: can't read superblock
    mount: RPC: Authentication error; why = Client credential too weak

I find these error messages suspicious, and do not believe either of
them represent the true problem.  I have appended the debug output in
case that is useful.  I notice that it seems to try the mount seven
times each, before giving up with one of the above errors (usually
it is the superblock error).

I have also noticed corruption of /etc/mtab.  Sometimes static mounts
just disappear from it during busy times.  I assume this is some type
of race condition where file locking fails.  I do not know if this is
the same problem or a different problem, but I thought I would mention it.

I do not know what causes such a high load of mount requests.  But I see
them in the accounting files.  Nothing obvious is going on when the errors
occur.  They happen for several seconds at a time, 6-8 times per day.

The NFS client is ubuntu 6.06 (dapper), kernel 2.6.17.4, automount 4.1.4.
The NFS server is a Network Appliances filer.

Thanks,
Fletcher

Nov  2 08:41:09 smtp automount[3862]: handle_child: got pid 15301, sig 0 (0), stat 0 
Nov  2 08:41:09 smtp automount[3862]: sig_child: found pending iop pid 15301: signalled 0 (sig 0), exit status 0
Nov  2 08:41:09 smtp automount[3862]: send_ready: token=4324 
Nov  2 08:41:11 smtp automount[3862]: handle_packet: type = 0 
Nov  2 08:41:11 smtp automount[3862]: handle_packet_missing: token 4325, name v2q007 
Nov  2 08:41:11 smtp automount[3862]: attempting to mount entry /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: lookup(yp): looking up v2q007
Nov  2 08:41:11 smtp automount[15326]: ret = 1
Nov  2 08:41:11 smtp automount[15326]: lookup(yp): v2q007 -> -intr,grpid,quota,retry=2^Ifiler3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: parse(sun): expanded entry: -intr,grpid,quota,retry=2^Ifiler3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: parse(sun): dequote("intr,grpid,quota,retry=2") -> intr,grpid,quota,retry=2
Nov  2 08:41:11 smtp automount[15326]: parse(sun): gathered options: tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2
Nov  2 08:41:11 smtp automount[15326]: parse(sun): dequote("filer3:/vol/vol2/v2q007") -> filer3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: parse(sun): core of entry: options=tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2, loc=filer3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: parse(sun): mounting root /v/filer3, mountpoint v2q007, what filer3:/vol/vol2/v2q007, fstype nfs, options tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2 
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): root=/v/filer3 name=v2q007 what=filer3:/vol/vol2/v2q007, fstype=nfs, options=tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): nfs options="tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2", nosymlink=0, symlink=0, ro=0
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): is_local_mount: filer3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): from filer3:/vol/vol2/v2q007 elected filer3:/vol/vol2/v2q007
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): calling mkdir_path /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): calling mount -t nfs -s  -o tcp,rsize=8192,wsize=8192,intr,grpid,quota,retry=2 filer3:/vol/vol2/v2q007 /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: >> mount: filer3:/vol/vol2/v2q007: can't read superblock
Nov  2 08:41:11 smtp automount[15326]: mount(nfs): nfs: mount failure filer3:/vol/vol2/v2q007 on /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: failed to mount /v/filer3/v2q007
Nov  2 08:41:11 smtp automount[15326]: umount_multi: path=/v/filer3/v2q007 incl=1 
Nov  2 08:41:11 smtp automount[15326]: umount_multi: no mounts found under /v/filer3/v2q007