Seeing some 5.0.1 stop expiring mounts

Seeing some 5.0.1 stop expiring mounts

[Mike Marion]

autofs-5.0.1 with patches:

autofs-5.0.1-bad-cast.patch
autofs-5.0.1-check-mtab-updated.patch
autofs-5.0.1-check-user-info-return.patch
autofs-5.0.1-cmd-global-options-fix.patch
autofs-5.0.1-cmd-global-options.patch
autofs-5.0.1-code-cleanups.patch
autofs-5.0.1-conf-append-global.patch
autofs-5.0.1-configure-cleanups.patch
autofs-5.0.1-correct-hesiod-check.patch
autofs-5.0.1-disable-exports-check.patch
autofs-5.0.1-drop-default-prefix-from-config.patch
autofs-5.0.1-export-check-network-fix-2.patch
autofs-5.0.1-file-map-allow-white-space-only-line.patch
autofs-5.0.1-fix-browse-dir-create.patch
autofs-5.0.1-hosts-simple-fail.patch
autofs-5.0.1-localfs-label-check.patch
autofs-5.0.1-map-update-source-only.patch
autofs-5.0.1-network_match-fix.patch
autofs-5.0.1-null-domain-fix.patch
autofs-5.0.1-random-selection.patch
autofs-5.0.1-remove-macro-automount-8.patch
autofs-5.0.1-remove-redundant-ident-macros.patch
autofs-5.0.1-update-kernel-patches.patch

kernel 2.6.16.21-0.9 on sles9-sp3 (took sles10 kernel src.rpm and built
new kernel on top of sles9).

All maps in LDAP.  Startup is fine... 6200+ entries in /proc/mounts (Yes
we have that many).  Some hosts get into a state where they aren't
expiring mounts anymore:
$ mount -t nfs | wc
    476    2856   70188

Differences I see in ps are that there's another thread hanging around
(wedged?):
ps axsm output:
    0  2840 0000000000010201                -                -
- -    ?        384:58 automoun    0     - 0000000000000000
  fffffffe7ffbfeff 0000000000000000 0000000180000000 Tsl  -
0:00 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:00 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:16 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:04 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:10 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:09 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:00 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:00 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:02 -

ps -eLF output:
root      2840     1  2840  0    9 15576  7208   2 Jul05 ?
00:00:00 automount
root      2840     1  2841  0    9 15576  7208   2 Jul05 ?
00:00:00 automount
root      2840     1  2842  0    9 15576  7208   2 Jul05 ?
00:00:16 automount
root      2840     1  2845  0    9 15576  7208   1 Jul05 ?
00:00:04 automount
root      2840     1  2848  0    9 15576  7208   0 Jul05 ?
00:00:10 automount
root      2840     1  2849  0    9 15576  7208   3 Jul05 ?
00:00:09 automount
root      2840     1 32417  0    9 15576  7208   2 Jul13 ?
00:00:00 automount
root      2840     1  2230  0    9 15576  7208   3 Jul14 ?
00:00:00 automount
root      2840     1  2235  0    9 15576  7208   1 Jul14 ?
00:00:02 automount

Hosts that are ok spawn a new thread periodically, the wedged hosts
don't.  Kill -USR1/HUP don't seem to have any effect, even stracing
process while sending that signal shows that it never seems to see it
(no SIGUSR1 info in strace).  

Strace shows host stuck doing nothing but futex and time calls like
this:
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949131, {0, 993887000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701903, 6010000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949133, {0, 993990000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701904, 5912000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949135, {0, 994088000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701905, 5758000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949137, {0, 994242000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701906, 5657000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949139, {0, 994343000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701907, 5625000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949141, {0, 994375000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701908, 5506000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949143, {0, 994494000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701909, 5422000}) = 0


Wondering if anyone else has seen and/or has any idea what might be the
cause.

-- 
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
"I've never used their tech support, but the word is that it sucks."
I believe it's composed entirely of monkeys that couldn't get the Shakespeare
gig. :-) ==> /. users talking about @home tech support

Re: Seeing some 5.0.1 stop expiring mounts

[Mike Marion]

On Tue, Jul 17, 2007 at 12:54:31PM -0700, Mike Marion wrote:

> Wondering if anyone else has seen and/or has any idea what might be the
> cause.

Hmm.. might've found the cause at least.. common thread appears to be
OOM situation hit on hosts in this state.  Yes, our users do this a
LOT.. it's a big pain, we're trying to find a balance between enforcing
limits without killing batch jobs incorrectly.

Luckily autofs5 seems to be the first autofs IME that you can kill -9
then start a new one and have it take over the mounts properly.

-- 
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
My pid is Inigo Montoya.  You kill -9 my parent process. Prepare to vi.

Re: Seeing some 5.0.1 stop expiring mounts

[Ian Kent]

On Tue, 2007-07-17 at 13:04 -0700, Mike Marion wrote:
> On Tue, Jul 17, 2007 at 12:54:31PM -0700, Mike Marion wrote:
> 
> > Wondering if anyone else has seen and/or has any idea what might be the
> > cause.
> 
> Hmm.. might've found the cause at least.. common thread appears to be
> OOM situation hit on hosts in this state.  Yes, our users do this a
> LOT.. it's a big pain, we're trying to find a balance between enforcing
> limits without killing batch jobs incorrectly.
> 
> Luckily autofs5 seems to be the first autofs IME that you can kill -9
> then start a new one and have it take over the mounts properly.

Yep, it's seems to work well so far.

If you configure with --enable-ignore-busy (and for good measure
--disable-mount-locking) then you should be able to achieve the same
result with "service autofs restart", no kill needed at all.

Ian

Re: Seeing some 5.0.1 stop expiring mounts

[Ian Kent]

On Tue, 2007-07-17 at 12:54 -0700, Mike Marion wrote:
> autofs-5.0.1 with patches:
> 
> autofs-5.0.1-bad-cast.patch
> autofs-5.0.1-check-mtab-updated.patch
> autofs-5.0.1-check-user-info-return.patch
> autofs-5.0.1-cmd-global-options-fix.patch
> autofs-5.0.1-cmd-global-options.patch
> autofs-5.0.1-code-cleanups.patch
> autofs-5.0.1-conf-append-global.patch
> autofs-5.0.1-configure-cleanups.patch
> autofs-5.0.1-correct-hesiod-check.patch
> autofs-5.0.1-disable-exports-check.patch
> autofs-5.0.1-drop-default-prefix-from-config.patch
> autofs-5.0.1-export-check-network-fix-2.patch
> autofs-5.0.1-file-map-allow-white-space-only-line.patch
> autofs-5.0.1-fix-browse-dir-create.patch
> autofs-5.0.1-hosts-simple-fail.patch
> autofs-5.0.1-localfs-label-check.patch
> autofs-5.0.1-map-update-source-only.patch
> autofs-5.0.1-network_match-fix.patch
> autofs-5.0.1-null-domain-fix.patch
> autofs-5.0.1-random-selection.patch
> autofs-5.0.1-remove-macro-automount-8.patch
> autofs-5.0.1-remove-redundant-ident-macros.patch
> autofs-5.0.1-update-kernel-patches.patch

Might be worth considering going to 5.0.2, especially since you have a
busy site, as a nasty deadlock in the alarm handler has been fixed. 

There are patches for 5.0.1 on kernel.org as well.

Ian

Re: Seeing some 5.0.1 stop expiring mounts

[Mike Marion]

On Wed, Jul 18, 2007 at 02:46:07PM +0800, Ian Kent wrote:

> Might be worth considering going to 5.0.2, especially since you have a
> busy site, as a nasty deadlock in the alarm handler has been fixed. 

Unfortunately, we can't.  I've since found out (through trial and error)
that the patch:
> > autofs-5.0.1-map-update-source-only.patch

is completely broken for us, and it appears to be part of the 5.0.2
codebase now.

Our main setup for auotofs5 clients now is pure ldap.. auto_master is in
ldap, nsswitch.conf has
automount: ldap
everything is in ldap.  We remove all /etc/auto.* files too.

With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the
daemon gets a HUP signal, it flushes out all the auto.projects (our main
map) entries from /proc/mounts and they're gone forever.

When first started, and until the daemon gets a HUP, it works fine.  Our
/proc/mounts has 6200+ entries (we have a crapload of paths) and
they'll mount great.  Entries look as expected, e.g.:
auto.projects /prj/qct/gv autofs rw,fd=6,pgrp=2571,timeout=600,minproto=5,maxproto=5,direct 0 0
then if you mount it, it adds in:
ronald:/vol/eng_ice_0014/qct_gv /prj/qct/gv nfs rw,v3,rsize=32768,wsize=32768,acregmin=1,acregmax=5,acdirmin=1,acdirmax=5,hard,lock,proto=tcp,addr=ronald 0 0

After the HUP, the thing flushes, then logs a ton of rm_dir errors..
like so:

Start of daemon:
automount[2571]: mounted direct mount on /prj/qct/gv with timeout 600, freq 150 seconds

Flush after HUP:
automount[2572]: umounted direct mount /prj/qct/gv

After all umounts.. these errors show for every path:
automount[2549]: rmdir_path: lstat of /prj/qct/gv failed.

I did a test with 5.0.1 with all patchs sans the
autofs-5.0.1-map-update-source-only.patch and it's fine... I can HUP
left and right, do kill -USR1 to flush, etc.  Works right.  But rebuild again
with that patch and first HUP breaks all our auto.projects paths.  Weird
thing is the /net and /usr2 (indirect home dirs) stay working.  Those
entries look like:
$ egrep 'auto.home|/net' /proc/mounts | grep -v auto.projects
-hosts /net autofs rw,fd=9,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0
auto.home /usr2 autofs rw,fd=14,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0

I can try to get more info out of the ldap setup if needed, but
unfortunately the admin that really knows the ldap side is out until
next week.  I can poke around some though, and use his docs.

-- 
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
The needs of the many outweigh the needs of the few, or the one.  That's from
Trek.  Pretty cool, huh? -- Dennis Miller

Re: Seeing some 5.0.1 stop expiring mounts

[Ian Kent]

On Wed, 2007-07-18 at 14:39 -0700, Mike Marion wrote:
> On Wed, Jul 18, 2007 at 02:46:07PM +0800, Ian Kent wrote:
> 
> > Might be worth considering going to 5.0.2, especially since you have a
> > busy site, as a nasty deadlock in the alarm handler has been fixed. 
> 
> Unfortunately, we can't.  I've since found out (through trial and error)
> that the patch:
> > > autofs-5.0.1-map-update-source-only.patch

Oops!

> 
> is completely broken for us, and it appears to be part of the 5.0.2
> codebase now.
> 
> Our main setup for auotofs5 clients now is pure ldap.. auto_master is in
> ldap, nsswitch.conf has
> automount: ldap
> everything is in ldap.  We remove all /etc/auto.* files too.
> 
> With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the
> daemon gets a HUP signal, it flushes out all the auto.projects (our main
> map) entries from /proc/mounts and they're gone forever.
> 
> When first started, and until the daemon gets a HUP, it works fine.  Our
> /proc/mounts has 6200+ entries (we have a crapload of paths) and
> they'll mount great.  Entries look as expected, e.g.:
> auto.projects /prj/qct/gv autofs rw,fd=6,pgrp=2571,timeout=600,minproto=5,maxproto=5,direct 0 0
> then if you mount it, it adds in:
> ronald:/vol/eng_ice_0014/qct_gv /prj/qct/gv nfs rw,v3,rsize=32768,wsize=32768,acregmin=1,acregmax=5,acdirmin=1,acdirmax=5,hard,lock,proto=tcp,addr=ronald 0 0
> 
> After the HUP, the thing flushes, then logs a ton of rm_dir errors..
> like so:
> 
> Start of daemon:
> automount[2571]: mounted direct mount on /prj/qct/gv with timeout 600, freq 150 seconds
> 
> Flush after HUP:
> automount[2572]: umounted direct mount /prj/qct/gv
> 
> After all umounts.. these errors show for every path:
> automount[2549]: rmdir_path: lstat of /prj/qct/gv failed.
> 
> I did a test with 5.0.1 with all patchs sans the
> autofs-5.0.1-map-update-source-only.patch and it's fine... I can HUP
> left and right, do kill -USR1 to flush, etc.  Works right.  But rebuild again
> with that patch and first HUP breaks all our auto.projects paths.  Weird
> thing is the /net and /usr2 (indirect home dirs) stay working.  Those
> entries look like:
> $ egrep 'auto.home|/net' /proc/mounts | grep -v auto.projects
> -hosts /net autofs rw,fd=9,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0
> auto.home /usr2 autofs rw,fd=14,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0

So it sounds like the direct mounts are unhappy with this or possibly
the LDAP module.

I'll try to duplicate the problem.

Ian

Re: Seeing some 5.0.1 stop expiring mounts

[Ian Kent]

On Wed, 2007-07-18 at 14:39 -0700, Mike Marion wrote:
> On Wed, Jul 18, 2007 at 02:46:07PM +0800, Ian Kent wrote:
> 
> > Might be worth considering going to 5.0.2, especially since you have a
> > busy site, as a nasty deadlock in the alarm handler has been fixed. 
> 
> Unfortunately, we can't.  I've since found out (through trial and error)
> that the patch:
> > > autofs-5.0.1-map-update-source-only.patch
> 
> is completely broken for us, and it appears to be part of the 5.0.2
> codebase now.

Indeed.

I think I've identified the problem but I'll need to look more closely
at it.

In the meantime it would be great if you could try this patch, in
addition to the patch above, and check if there are any other problems
with it.

The patch is against the current git repo but applies against a fairly
up to date 5.0.1 tree (it looks correct) with a small offset.

Ian

---
diff --git a/daemon/lookup.c b/daemon/lookup.c
index 70b9e02..4f2b318 100644
--- a/daemon/lookup.c
+++ b/daemon/lookup.c
@@ -325,6 +325,7 @@ static int read_file_source_instance(struct autofs_point *ap, struct map_source
 		instance->recurse = map->recurse;
 		instance->depth = map->depth;
 	}
+	instance->stale = map->stale;
 
 	return do_read_map(ap, instance, age);
 }
@@ -346,6 +347,7 @@ static int read_source_instance(struct autofs_point *ap, struct map_source *map,
 		instance->recurse = map->recurse;
 		instance->depth = map->depth;
 	}
+	instance->stale = map->stale;
 
 	return do_read_map(ap, instance, age);
 }

Re: Seeing some 5.0.1 stop expiring mounts

[Ian Kent]

On Wed, 2007-07-18 at 14:39 -0700, Mike Marion wrote:
> On Wed, Jul 18, 2007 at 02:46:07PM +0800, Ian Kent wrote:
> 
> > Might be worth considering going to 5.0.2, especially since you have a
> > busy site, as a nasty deadlock in the alarm handler has been fixed. 
> 

snip ...

> With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the
> daemon gets a HUP signal, it flushes out all the auto.projects (our main
> map) entries from /proc/mounts and they're gone forever.

fwiw, the map-update-source-only patch is quite an important change from
the point of view of the version 5 design and the correct operation of
the map update logic. The reason it was needed is that I changed the way
instances of maps are sorted during initial development but didn't
properly account for it.

I really appreciate any time you can spare to help with this.

Thanks
Ian

Re: Seeing some 5.0.1 stop expiring mounts

[Mike Marion]

On Thu, Jul 19, 2007 at 08:12:19PM +0800, Ian Kent wrote:

> I really appreciate any time you can spare to help with this.

No problem.  I'm finishing up some stuff needed today but should have
time to try out the other patch and see if it still has that expire
problem tomorrow.

-- 
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
Girl: "She dumped ya, didn't she?"
Drew: "It was mutual.  We both hated me, and she was the only one that had 
nerve enough to leave."  ==> Drew Cary Show

Re: Seeing some 5.0.1 stop expiring mounts

[Ian Kent]

On Thu, 2007-07-19 at 16:40 -0700, Mike Marion wrote:
> On Thu, Jul 19, 2007 at 08:12:19PM +0800, Ian Kent wrote:
> 
> > I really appreciate any time you can spare to help with this.
> 
> No problem.  I'm finishing up some stuff needed today but should have
> time to try out the other patch and see if it still has that expire
> problem tomorrow.

Don't mean to be a pain but have you had time to check this?

Ian

Re: Seeing some 5.0.1 stop expiring mounts

[Mike Marion]

On Fri, Jul 27, 2007 at 11:31:20AM +0800, Ian Kent wrote:

> Don't mean to be a pain but have you had time to check this?

Not a pain.. too much got in the way this week. I was able to test that
patch both against 5.0.1 with the
autofs-5.0.1-map-update-source-only.patch 
and against 5.0.2 and it's good.

Just to show, here's 5.0.2 without those 2 lines patched in2

vbr2-001:/usr/src/packages/SOURCES # ps auxw | grep automount
root      6812  0.7  0.0  28552  7180 ?        Ssl  13:59   0:00
automount
root      6935  0.0  0.0   2888   704 pts/1    S+   14:00   0:00 grep
automount
vbr2-001:/usr/src/packages/SOURCES # wc /proc/mounts; mount | wc ; kill
-HUP 6812; while [ 1 ]; do wc /proc/mounts ; mount | wc; sleep 5; done
6314 37884 730725 /proc/mounts
     12      72     441
6314 37884 730725 /proc/mounts
     12      72     441
6107 36642 706276 /proc/mounts
     12      72     441
5964 35784 689905 /proc/mounts
     12      72     441
5833 34998 674567 /proc/mounts
     12      72     441
5695 34170 658506 /proc/mounts
     12      72     441
5477 32862 633119 /proc/mounts
     12      72     441


and on down until all the auto.projects entries are gone... which takes
awhile.

The syslog shows them all being unmounted, unlike 5.0.1 it only shows
the "rmdir lmstat of <path> failed" messages for things that were
actually nfs mounted vs the entire auto.projects tree though.

-- 
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
Lisa: "As you know, we've been swimming. And we've developed a taste for it. We
agree that getting our own pool is the way to go. Now before you respond, you 
should know that your refusal will result in months and months of..."
Bart, Lisa: "CanwehaveapoolDad? CanwehaveapoolDad? CanwehaveapoolDad? 
CanwehaveapoolDad? CanwehaveapoolDad?"
Homer: "I understand. Let us celebrate our agreement with the adding of chocolate to milk." -- Simpsons

Re: Seeing some 5.0.1 stop expiring mounts

[Ian Kent]

On Fri, 2007-07-27 at 15:06 -0700, Mike Marion wrote:
> On Fri, Jul 27, 2007 at 11:31:20AM +0800, Ian Kent wrote:
> 
> > Don't mean to be a pain but have you had time to check this?
> 
> Not a pain.. too much got in the way this week. I was able to test that
> patch both against 5.0.1 with the
> autofs-5.0.1-map-update-source-only.patch 
> and against 5.0.2 and it's good.

Cool .. I'll commit the patch to the repo and deal with any other issues
as we need this anyway.

Thanks

> 
> Just to show, here's 5.0.2 without those 2 lines patched in2
> 
> vbr2-001:/usr/src/packages/SOURCES # ps auxw | grep automount
> root      6812  0.7  0.0  28552  7180 ?        Ssl  13:59   0:00
> automount
> root      6935  0.0  0.0   2888   704 pts/1    S+   14:00   0:00 grep
> automount
> vbr2-001:/usr/src/packages/SOURCES # wc /proc/mounts; mount | wc ; kill
> -HUP 6812; while [ 1 ]; do wc /proc/mounts ; mount | wc; sleep 5; done
> 6314 37884 730725 /proc/mounts
>      12      72     441
> 6314 37884 730725 /proc/mounts
>      12      72     441
> 6107 36642 706276 /proc/mounts
>      12      72     441
> 5964 35784 689905 /proc/mounts
>      12      72     441
> 5833 34998 674567 /proc/mounts
>      12      72     441
> 5695 34170 658506 /proc/mounts
>      12      72     441
> 5477 32862 633119 /proc/mounts
>      12      72     441
> 
> 
> and on down until all the auto.projects entries are gone... which takes
> awhile.
> 
> The syslog shows them all being unmounted, unlike 5.0.1 it only shows
> the "rmdir lmstat of <path> failed" messages for things that were
> actually nfs mounted vs the entire auto.projects tree though.
>