Severe problem with 4.1.2: automount caches NIS maps forever

Severe problem with 4.1.2: automount caches NIS maps forever

[Bryan O'Sullivan]

Hi -

The autofs 4.1.2 release has a change in behaviour from earlier releases
(4.1.0, I think, and 4.0.0 for sure) in that it never checks to see if a
NIS-based automount map has changed.  This means that if I change the
location of an autofs-mounted filesystem, I have to SIGHUP the automount
daemons on every machine that care about that filesystem so that they'll
forget the contents of the map and reload it.

Needless to say, this doesn't scale in even a tiny way :-(

	<b

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[mmarion]

On  6 May, Bryan O'Sullivan wrote:

> The autofs 4.1.2 release has a change in behaviour from earlier releases
> (4.1.0, I think, and 4.0.0 for sure) in that it never checks to see if a
> NIS-based automount map has changed.  This means that if I change the
> location of an autofs-mounted filesystem, I have to SIGHUP the automount
> daemons on every machine that care about that filesystem so that they'll
> forget the contents of the map and reload it.

I noticed this too and did a workaround...

I wrote a script that runs every 15 minutes out of cron and does an md5sum of
the current yp map, compares to the last known md5sum (saved in a file in
/etc) and if different, does a kill -HUP on the automount daemon.  This at
least means that changes will take no more then 15 minutes to be seen, and the
daemon won't just be hit with SIGHUPs when they aren't needed.

But yeah, it'd be nice if the daemon could do it's own periodic checks for
changes... and if that interval were configurable, even better.

-- 
Mike Marion-Unix SysAdmin/Staff Engineer-http://www.qualcomm.com
Marge: "Homer, sitting that close to the TV can't be good for you."
Homer: "Talking while the TV's on can't be good for you!"
==> Simpsons

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Tom Georgoulias]

mmarion@qualcomm.com wrote:
> On  6 May, Bryan O'Sullivan wrote:
>>The autofs 4.1.2 release has a change in behaviour from earlier releases
>>(4.1.0, I think, and 4.0.0 for sure) in that it never checks to see if a
>>NIS-based automount map has changed.

> I noticed this too and did a workaround...

Why was the behavor changed between releases?  I was asking about this 
in my other message and now I see that two other people can confirm the 
need to HUP for daemons to use new map changes.

Tom
-- 
Tom Georgoulias
POPI Classification
[x] General Business Information
[] Freescale Semiconductor Internal Use
[] Freescale Semiconductor Confidential Proprietary

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Michael Blandford]

Bryan O'Sullivan wrote:

>Hi -
>
>The autofs 4.1.2 release has a change in behaviour from earlier releases
>(4.1.0, I think, and 4.0.0 for sure) in that it never checks to see if a
>NIS-based automount map has changed.  This means that if I change the
>location of an autofs-mounted filesystem, I have to SIGHUP the automount
>daemons on every machine that care about that filesystem so that they'll
>forget the contents of the map and reload it.
>
>Needless to say, this doesn't scale in even a tiny way :-(
>  
>
Agreed.  You would think with all the testing I have done, I would have 
seen this one.

I can easily replicate it and fix it with a SIGHUP.

Ian, do you have any thoughts on this one?

Michael



Disclaimer: The content of this message is my personal opinion only and 
although I am an employee of Intel, the statements I make here in no way 
represent Intel's position on the issue, nor am I authorized to speak on 
behalf of Intel on this matter.

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Ian Kent]

On Thu, 6 May 2004, Michael Blandford wrote:

> Bryan O'Sullivan wrote:
> 
> >Hi -
> >
> >The autofs 4.1.2 release has a change in behaviour from earlier releases
> >(4.1.0, I think, and 4.0.0 for sure) in that it never checks to see if a
> >NIS-based automount map has changed.  This means that if I change the
> >location of an autofs-mounted filesystem, I have to SIGHUP the automount
> >daemons on every machine that care about that filesystem so that they'll
> >forget the contents of the map and reload it.
> >
> >Needless to say, this doesn't scale in even a tiny way :-(
> >  
> >
> Agreed.  You would think with all the testing I have done, I would have 
> seen this one.
> 
> I can easily replicate it and fix it with a SIGHUP.
> 
> Ian, do you have any thoughts on this one?

Yes.

This changed in 4.1.0 because, to do browsable directoties the entire map 
must be known in advance.

It is wrong to say the daemon never checks any more. It never did and to 
do so I need information about NIS and LDAP that I don't have at this 
stage.

In early stages I kept the "discover at mount time" behaviour but was 
concerned that this lead to a missleading picture of the mount tree.

So guys, the options here are:

1) leave as is and use HUP signal to refresh maps.
2) reintroduce the "discover at mount time" behaviour (possibly
   with a periodic resync).
3) add a periodic map refresh whose frequency is perhaps configurable.

I must retain the cache that is used internally as when (if I ever) get to 
working on fixing the current limitations the design will be heavily based 
on that.
 
Ian

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Jim Carter]

On Fri, 7 May 2004, Ian Kent wrote:
> This changed in 4.1.0 because, to do browsable directoties the entire map
> must be known in advance...
>
> So guys, the options here are:
>
> 1) leave as is and use HUP signal to refresh maps.
> 2) reintroduce the "discover at mount time" behaviour (possibly
>    with a periodic resync).
> 3) add a periodic map refresh whose frequency is perhaps configurable.

Another idea: if the map row is in the cache AND the mount succeeds,
fine, leave it in the cache.  If either one fails, the problem
might be with the map -- we've cached a row that was removed (and so was
its server), or we've failed to cache a newly added row.  So flush the
cache (for that map) and enumerate the map again.  But if a second mount
attempt fails, believe in the failure.  I think it likely that you can
get away with reading (or trying and failing to read) just that one row,
with random access data sources.

The only fly in the ointment is, if the mount options change, you won't
know.  I'd say, include a time to live (with a configurable timeout), so
if a row is needed that hasn't been read authoritatively in (let's say) one
hour, you should re-read just that one row, for random-access data sources
like NIS or LDAP or a programmatic map (or read the whole map if it's a
file).  Again, for ghost mounts, if someone does "ls /net/$host" and a full
enumeration hasn't been done within the timeout, you have to do it over.

I believe you said that wildcard maps are still going to work OK.  That's
important to me -- all my maps are wildcard maps.

The current HUP behavior needs to be retained if the sysop wants to update
an entire map immediately, but as one of the writers said, HUP is not very
scalable.  I have an automated script for doing this kind of thing, but if
there were 1000 machines it would still be a burden.

I think periodic refresh is not such a good idea because (at least in my
case) each machine can potentially use a whole lot of maps, but each one
has a few favorites, and it's a waste to refresh the others.  Refresh when
the data is requested and is stale.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Ian Kent]

On Thu, 6 May 2004, Jim Carter wrote:

> On Fri, 7 May 2004, Ian Kent wrote:
> > This changed in 4.1.0 because, to do browsable directoties the entire map
> > must be known in advance...
> >
> > So guys, the options here are:
> >
> > 1) leave as is and use HUP signal to refresh maps.
> > 2) reintroduce the "discover at mount time" behaviour (possibly
> >    with a periodic resync).
> > 3) add a periodic map refresh whose frequency is perhaps configurable.
> 
> Another idea: if the map row is in the cache AND the mount succeeds,
> fine, leave it in the cache.  If either one fails, the problem
> might be with the map -- we've cached a row that was removed (and so was
> its server), or we've failed to cache a newly added row.  So flush the
> cache (for that map) and enumerate the map again.  But if a second mount
> attempt fails, believe in the failure.  I think it likely that you can
> get away with reading (or trying and failing to read) just that one row,
> with random access data sources.

This is pretty much option 2. Your refresh ideas are a useful addition to 
the thought process. That's always been the hardest to detail in terms of 
operation.

> 
> The only fly in the ointment is, if the mount options change, you won't
> know.  I'd say, include a time to live (with a configurable timeout), so
> if a row is needed that hasn't been read authoritatively in (let's say) one
> hour, you should re-read just that one row, for random-access data sources
> like NIS or LDAP or a programmatic map (or read the whole map if it's a
> file).  Again, for ghost mounts, if someone does "ls /net/$host" and a full
> enumeration hasn't been done within the timeout, you have to do it over.

Mmm. above.

> 
> I believe you said that wildcard maps are still going to work OK.  That's
> important to me -- all my maps are wildcard maps.

Still do. Must continue to do so. Otherwise its got to be fixed.

> 
> The current HUP behavior needs to be retained if the sysop wants to update
> an entire map immediately, but as one of the writers said, HUP is not very
> scalable.  I have an automated script for doing this kind of thing, but if
> there were 1000 machines it would still be a burden.

That should be fine. It's likely more work to remove it anyway.

> 
> I think periodic refresh is not such a good idea because (at least in my
> case) each machine can potentially use a whole lot of maps, but each one
> has a few favorites, and it's a waste to refresh the others.  Refresh when
> the data is requested and is stale.

Understand your point and agree.

Ian

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Michael Blandford]

Jim Carter wrote:

>On Fri, 7 May 2004, Ian Kent wrote:
>  
>
>>This changed in 4.1.0 because, to do browsable directoties the entire map
>>must be known in advance...
>>
>>So guys, the options here are:
>>
>>1) leave as is and use HUP signal to refresh maps.
>>
I don't think this solution can work.  It puts to much burden on the 
clients.  It would be impossible to get consistency across a large 
number of clients.

>>2) reintroduce the "discover at mount time" behaviour (possibly
>>   with a periodic resync).
>>
Does this mean verify the map entry once again at the time of mount?  If 
so, that seems like it might work.

>>3) add a periodic map refresh whose frequency is perhaps configurable.
>>    
>>
I think a periodic map refresh +  verify the map entry at mount time 
could resolve the issue.

map refreshes could be done on a random schedule within a time.  That 
would keep the load down on the map servers.

The random map refresh could keep the ghosting data fairly valid while 
mounts would be 100% consistent.

>Another idea: if the map row is in the cache AND the mount succeeds,
>fine, leave it in the cache.  If either one fails, the problem
>might be with the map -- we've cached a row that was removed (and so was
>its server), or we've failed to cache a newly added row.  So flush the
>cache (for that map) and enumerate the map again.  But if a second mount
>attempt fails, believe in the failure.  I think it likely that you can
>get away with reading (or trying and failing to read) just that one row,
>with random access data sources.
>  
>
What happens if you change the mount point but the old remains?  autofs 
wouldn't notice the change in your scenario and we would get the wrong 
behaviour.

It seems like we need to verify the cached entry at every mount.

>The only fly in the ointment is, if the mount options change, you won't
>know.  I'd say, include a time to live (with a configurable timeout), so
>if a row is needed that hasn't been read authoritatively in (let's say) one
>hour, you should re-read just that one row, for random-access data sources
>
I just don't think that works well enough.  If a map changes, autofs 
needs to notice.  Waiting an hour or even 5 minutes isn't the right 
solution. 

>The current HUP behavior needs to be retained if the sysop wants to update
>an entire map immediately, but as one of the writers said, HUP is not very
>scalable.  I have an automated script for doing this kind of thing, but if
>there were 1000 machines it would still be a burden.
>  
>
Think of even larger deployments.  HUP'ing isnt a just a burden, it is 
impractical.

Michael

Disclaimer: The content of this message is my personal opinion only and 
although I am an employee of Intel, the statements I make here in no way 
represent Intel's position on the issue, nor am I authorized to speak on 
behalf of Intel on this matter.

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[raven]

On Fri, 7 May 2004, Michael Blandford wrote:

> Jim Carter wrote:
> 
> >On Fri, 7 May 2004, Ian Kent wrote:
> >  
> >
> >>This changed in 4.1.0 because, to do browsable directoties the entire map
> >>must be known in advance...
> >>
> >>So guys, the options here are:
> >>
> >>1) leave as is and use HUP signal to refresh maps.
> >>
> I don't think this solution can work.  It puts to much burden on the 
> clients.  It would be impossible to get consistency across a large 
> number of clients.

Yes. We must be able to do better than this.

> 
> >>2) reintroduce the "discover at mount time" behaviour (possibly
> >>   with a periodic resync).
> >>
> Does this mean verify the map entry once again at the time of mount?  If 
> so, that seems like it might work.

Basically yes but I thought having a time after which the entry is 
considered potentially stale would be OK. Say, 1/4 of the timeout.

> 
> >>3) add a periodic map refresh whose frequency is perhaps configurable.
> >>    
> >>
> I think a periodic map refresh +  verify the map entry at mount time 
> could resolve the issue.
>

The main reason I have avoided this is because of potential problems 
accessing the map server. If your NIS server is not available, for 
some reason, but your NFS servers are OK then autofs comes to a grinding 
halt. NIS in particular is problematic like this.
 
> map refreshes could be done on a random schedule within a time.  That 
> would keep the load down on the map servers.
> 
> The random map refresh could keep the ghosting data fairly valid while 
> mounts would be 100% consistent.
> 
> >Another idea: if the map row is in the cache AND the mount succeeds,
> >fine, leave it in the cache.  If either one fails, the problem
> >might be with the map -- we've cached a row that was removed (and so was
> >its server), or we've failed to cache a newly added row.  So flush the
> >cache (for that map) and enumerate the map again.  But if a second mount
> >attempt fails, believe in the failure.  I think it likely that you can
> >get away with reading (or trying and failing to read) just that one row,
> >with random access data sources.
> >  
> >
> What happens if you change the mount point but the old remains?  autofs 
> wouldn't notice the change in your scenario and we would get the wrong 
> behaviour.

Good point.

The new entry gets added to the map. Leaving the original as well.
Not good. I'll have to think about that.

> 
> It seems like we need to verify the cached entry at every mount.
> 
> >The only fly in the ointment is, if the mount options change, you won't
> >know.  I'd say, include a time to live (with a configurable timeout), so
> >if a row is needed that hasn't been read authoritatively in (let's say) one
> >hour, you should re-read just that one row, for random-access data sources
> >
> I just don't think that works well enough.  If a map changes, autofs 
> needs to notice.  Waiting an hour or even 5 minutes isn't the right 
> solution. 
> 
> >The current HUP behavior needs to be retained if the sysop wants to update
> >an entire map immediately, but as one of the writers said, HUP is not very
> >scalable.  I have an automated script for doing this kind of thing, but if
> >there were 1000 machines it would still be a burden.
> >  
> >
> Think of even larger deployments.  HUP'ing isnt a just a burden, it is 
> impractical.

Should be retained anyway, in addition to the dynamic changes.

Ian

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Jim Carter]

On Sat, 8 May 2004 raven@themaw.net wrote:

> > I think a periodic map refresh +  verify the map entry at mount time
> > could resolve the issue.
> >
>
> The main reason I have avoided this is because of potential problems
> accessing the map server. If your NIS server is not available, for
> some reason, but your NFS servers are OK then autofs comes to a grinding
> halt. NIS in particular is problematic like this.

Good point here; it's happened to us.  So specify a time-to-live.

1.  If the needed map row is in the cache AND is younger than the TTL,
    use it as-is.  Deletions from the map, or changed mount options, will
    not be seen until the TTL  expires.
2.  Otherwise, read the map row (needing YP for some maps).  This might
    fail, because the server is dead or the row is no longer in the map.
    You need to distinguish these cases; wipe the cache row for the latter.
3.  If now the row is in the cache (either from YP or a stale cache entry),
    use it.
4.  If the row is not in the cache, you lose.

Then you go on to the conditional on whether the mount works:

5.  If the requested filesystem can be mounted, you're done.
6.  If it failed to mount, and this is the first try, the problem might be
    with the map rather than the server, so toss the cached map row and
    retry from step 2.
7.  If this is the 2nd try at mounting, you lose.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Bryan O'Sullivan]

On Thu, 2004-05-06 at 19:10, Ian Kent wrote:

> It is wrong to say the daemon never checks any more. It never did and to 
> do so I need information about NIS and LDAP that I don't have at this 
> stage.

In that case, I'd suggest that you put a time-to-live value on each
map.  Permit a configuration option to vary the value, but the default
can be something long, like an hour.

When automount reads a map, it sets the TTL for that map to a random
value between some lower limit (say TTL/4) and the configured or default
TTL.  This ensures that you won't get periodic storms of map-reading
network activity after a globally synchronous event like power
restoration or an automated package update (like a nightly yum update).

If a mount fails, you have an option: should you invalidate the cache
for that map, or not?  I don't have a good answer there.  The mount may
have failed because the map got updated, but it may also have failed
because the server was misconfigured, and you have no way of knowing
which.  Either way, the consequences are annoying: you run the risk of
software failures on the one hand, or excessive reads of automount maps
on the other.

One thing you could do in such a case is halve the remaining TTL on each
mount failure, to at least bring forward the next time the map will be
read.

	<b

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[raven]

On Fri, 7 May 2004, Bryan O'Sullivan wrote:

> On Thu, 2004-05-06 at 19:10, Ian Kent wrote:
> 
> > It is wrong to say the daemon never checks any more. It never did and to 
> > do so I need information about NIS and LDAP that I don't have at this 
> > stage.
> 
> In that case, I'd suggest that you put a time-to-live value on each
> map.  Permit a configuration option to vary the value, but the default
> can be something long, like an hour.
> 
> When automount reads a map, it sets the TTL for that map to a random
> value between some lower limit (say TTL/4) and the configured or default
> TTL.  This ensures that you won't get periodic storms of map-reading
> network activity after a globally synchronous event like power
> restoration or an automated package update (like a nightly yum update).

Maybe. As I've said I want to limit reading the entire map to a minimum.
But I would need to re-read the map sometime to clean the stale entries 
from "at mount time" changes.

> 
> If a mount fails, you have an option: should you invalidate the cache
> for that map, or not?  I don't have a good answer there.  The mount may
> have failed because the map got updated, but it may also have failed
> because the server was misconfigured, and you have no way of knowing
> which.  Either way, the consequences are annoying: you run the risk of
> software failures on the one hand, or excessive reads of automount maps
> on the other.

Dealing with mount fails is often a source of pain for a bunch of reasons. 
I think the idea of using that as a trigger to re-read the map is likely 
not the best as this is potentially the time the map might not be available.

Ian

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[Bryan O'Sullivan]

On Fri, 2004-05-07 at 22:44, raven@themaw.net wrote:

> Dealing with mount fails is often a source of pain for a bunch of reasons. 
> I think the idea of using that as a trigger to re-read the map is likely 
> not the best as this is potentially the time the map might not be available.

Right, which is why I suggest dropping the TTL instead of taking
immediate action.  Frankly, if NIS or LDAP is hosed and you're relying
on them for directory services, it is unlikely to matter whether or not
autofs "works", since so much else (i.e. regular user stuff) is going to
be hosed.

	<b

Re: Severe problem with 4.1.2: automount caches NIS maps forever

[raven]

On Sat, 8 May 2004, Bryan O'Sullivan wrote:

> On Fri, 2004-05-07 at 22:44, raven@themaw.net wrote:
> 
> > Dealing with mount fails is often a source of pain for a bunch of reasons. 
> > I think the idea of using that as a trigger to re-read the map is likely 
> > not the best as this is potentially the time the map might not be available.
> 
> Right, which is why I suggest dropping the TTL instead of taking
> immediate action.  Frankly, if NIS or LDAP is hosed and you're relying
> on them for directory services, it is unlikely to matter whether or not
> autofs "works", since so much else (i.e. regular user stuff) is going to
> be hosed.

Fair call.

But I don't want to have to deal with the "autofs is not working" calls 
(cause someones NIS is broke). Not everyone has stable infrastructure.

The other problem with a long timeout is that potentially incorrect map 
info is around for a long time. This part of the original complaint.

I can't help but think that the Solaris automounter doesn't do this for a 
good reason?

Ian