ldap and reloading

ldap and reloading

[Stef Bon]

Hello,

when using static file maps, or even executable maps, when the map 
changes, you'll have to reload the daemon to make these changes effective.

How does this work when all the data is in ldap? Does the automounter 
still creates a sort of snapshot of the map in memory, and to reread the 
data provided by ldap, a reload is neccesary? ldap is well know for that 
this is not needed. For example with postfix, it can use static maps, 
and when something changes, it has te be restarted. But when the lookup 
data (for local users for example) is in ldap, this is not neccesary.

If this is the case, I go directly over to ldap.

Stef

Re: ldap and reloading

[Ian Kent]

Stef Bon wrote:
> Hello,
> 
> when using static file maps, or even executable maps, when the map 
> changes, you'll have to reload the daemon to make these changes effective.
> 
> How does this work when all the data is in ldap? Does the automounter 
> still creates a sort of snapshot of the map in memory, and to reread the 
> data provided by ldap, a reload is neccesary? ldap is well know for that 
> this is not needed. For example with postfix, it can use static maps, 
> and when something changes, it has te be restarted. But when the lookup 
> data (for local users for example) is in ldap, this is not neccesary.

That's not quite right.

If you use the "browse" option then the entire map must be read in at
start. If not then autofs remembers entries that it has seen and
attempts to check their currency at lookup.

Each lookup should check if the entry is still up to date and attempts
to work out if the map has changed (although it's not quite as simple as
that). If we think the map has changed a re-load should be triggered
internally. Following the (or any) re-load there is a cleanup which is
probably why it looks like map changes aren't seen. Any changes in
multi-mount entries cannot be seen until after they have expired away,
because of the need to maintain the context of the entry over the
duration of the mount.

Direct maps don't quite do this properly, partly because of the way they
work and partly because of an issue I haven't addressed yet.

Clearly, with program maps, we need to rely on the re-load to a large
extent but a best effort is made to work out if the entry is stale,
however, we just don't have anything really to use to establish this, so
 a re-load is needed to clean them up.

Ian

Re: ldap and reloading

[Stef Bon]

Ian Kent wrote:
> Stef Bon wrote:
>   
>> Hello,
>>
>> when using static file maps, or even executable maps, when the map 
>> changes, you'll have to reload the daemon to make these changes effective.
>>
>> How does this work when all the data is in ldap? Does the automounter 
>> still creates a sort of snapshot of the map in memory, and to reread the 
>> data provided by ldap, a reload is neccesary? ldap is well know for that 
>> this is not needed. For example with postfix, it can use static maps, 
>> and when something changes, it has te be restarted. But when the lookup 
>> data (for local users for example) is in ldap, this is not neccesary.
>>     
>
> That's not quite right.
>
> If you use the "browse" option then the entire map must be read in at
> start. If not then autofs remembers entries that it has seen and
> attempts to check their currency at lookup.
>
> Each lookup should check if the entry is still up to date and attempts
> to work out if the map has changed (although it's not quite as simple as
> that). If we think the map has changed a re-load should be triggered
> internally. Following the (or any) re-load there is a cleanup which is
> probably why it looks like map changes aren't seen. Any changes in
> multi-mount entries cannot be seen until after they have expired away,
> because of the need to maintain the context of the entry over the
> duration of the mount.
>
> Direct maps don't quite do this properly, partly because of the way they
> work and partly because of an issue I haven't addressed yet.
>
> Clearly, with program maps, we need to rely on the re-load to a large
> extent but a best effort is made to work out if the entry is stale,
> however, we just don't have anything really to use to establish this, so
>  a re-load is needed to clean them up.
>   

Ok Ian,

maybe you're right. You probably are, I cannot discuss this with you, 
you're the expert.

You say that it may look like map changes aren't seen. How, as developer 
of a construction making use of autofs,
can I see/check it's reloaded. Are there any indications, or triggers 
which always lead to this internal reloading?
BTW I also use reloading when a map/mountpoint is added or removed, so 
not only when a map changes.

I know multimount entries are difficult, and are handled as one.

But how about ldap? I assume you're talking about how maps are handled 
here in general, but is this different/the same with ldap??

Stef Bon

Re: ldap and reloading

[Ian Kent]

Stef Bon wrote:
> Ian Kent wrote:
>> Stef Bon wrote:
>>  
>>> Hello,
>>>
>>> when using static file maps, or even executable maps, when the map
>>> changes, you'll have to reload the daemon to make these changes
>>> effective.
>>>
>>> How does this work when all the data is in ldap? Does the automounter
>>> still creates a sort of snapshot of the map in memory, and to reread
>>> the data provided by ldap, a reload is neccesary? ldap is well know
>>> for that this is not needed. For example with postfix, it can use
>>> static maps, and when something changes, it has te be restarted. But
>>> when the lookup data (for local users for example) is in ldap, this
>>> is not neccesary.
>>>     
>>
>> That's not quite right.
>>
>> If you use the "browse" option then the entire map must be read in at
>> start. If not then autofs remembers entries that it has seen and
>> attempts to check their currency at lookup.
>>
>> Each lookup should check if the entry is still up to date and attempts
>> to work out if the map has changed (although it's not quite as simple as
>> that). If we think the map has changed a re-load should be triggered
>> internally. Following the (or any) re-load there is a cleanup which is
>> probably why it looks like map changes aren't seen. Any changes in
>> multi-mount entries cannot be seen until after they have expired away,
>> because of the need to maintain the context of the entry over the
>> duration of the mount.
>>
>> Direct maps don't quite do this properly, partly because of the way they
>> work and partly because of an issue I haven't addressed yet.
>>
>> Clearly, with program maps, we need to rely on the re-load to a large
>> extent but a best effort is made to work out if the entry is stale,
>> however, we just don't have anything really to use to establish this, so
>>  a re-load is needed to clean them up.
>>   
> 
> Ok Ian,
> 
> maybe you're right. You probably are, I cannot discuss this with you,
> you're the expert.
> 
> You say that it may look like map changes aren't seen. How, as developer
> of a construction making use of autofs,
> can I see/check it's reloaded. Are there any indications, or triggers
> which always lead to this internal reloading?
> BTW I also use reloading when a map/mountpoint is added or removed, so
> not only when a map changes.

OK, but I think we're not talking about the same thing.

I always think about map entries when this come up but I suspect you are
thinking of master map entries which are different. There are no lookups
for the master map and so there is no opportunity to check if the map
has changed. To see changes to the master map a re-load is needed.

> 
> I know multimount entries are difficult, and are handled as one.
> 
> But how about ldap? I assume you're talking about how maps are handled
> here in general, but is this different/the same with ldap??

As far a the master map goes there isn't any real difference which
source is used.

As far as map entries go LDAP is probably the worst since it has no
standard, consistent information about when it was last updated
regardless of whether those entries are the master map or map entries
themselves.

Ian

Re: ldap and reloading

[Ondrej Valousek]

Talking about ldap maps reloading - maybe it is worth mentioning that if 
you configure automounter using the recently added DNS SRV support, 
automounter reloads itself as per zone TTL in DNS (that includes the 
master map, too) - right Ian?
Ondrej

Ian Kent wrote:
> Stef Bon wrote:
>   
>> Ian Kent wrote:
>>     
>>> Stef Bon wrote:
>>>  
>>>       
>>>> Hello,
>>>>
>>>> when using static file maps, or even executable maps, when the map
>>>> changes, you'll have to reload the daemon to make these changes
>>>> effective.
>>>>
>>>> How does this work when all the data is in ldap? Does the automounter
>>>> still creates a sort of snapshot of the map in memory, and to reread
>>>> the data provided by ldap, a reload is neccesary? ldap is well know
>>>> for that this is not needed. For example with postfix, it can use
>>>> static maps, and when something changes, it has te be restarted. But
>>>> when the lookup data (for local users for example) is in ldap, this
>>>> is not neccesary.
>>>>     
>>>>         
>>> That's not quite right.
>>>
>>> If you use the "browse" option then the entire map must be read in at
>>> start. If not then autofs remembers entries that it has seen and
>>> attempts to check their currency at lookup.
>>>
>>> Each lookup should check if the entry is still up to date and attempts
>>> to work out if the map has changed (although it's not quite as simple as
>>> that). If we think the map has changed a re-load should be triggered
>>> internally. Following the (or any) re-load there is a cleanup which is
>>> probably why it looks like map changes aren't seen. Any changes in
>>> multi-mount entries cannot be seen until after they have expired away,
>>> because of the need to maintain the context of the entry over the
>>> duration of the mount.
>>>
>>> Direct maps don't quite do this properly, partly because of the way they
>>> work and partly because of an issue I haven't addressed yet.
>>>
>>> Clearly, with program maps, we need to rely on the re-load to a large
>>> extent but a best effort is made to work out if the entry is stale,
>>> however, we just don't have anything really to use to establish this, so
>>>  a re-load is needed to clean them up.
>>>   
>>>       
>> Ok Ian,
>>
>> maybe you're right. You probably are, I cannot discuss this with you,
>> you're the expert.
>>
>> You say that it may look like map changes aren't seen. How, as developer
>> of a construction making use of autofs,
>> can I see/check it's reloaded. Are there any indications, or triggers
>> which always lead to this internal reloading?
>> BTW I also use reloading when a map/mountpoint is added or removed, so
>> not only when a map changes.
>>     
>
> OK, but I think we're not talking about the same thing.
>
> I always think about map entries when this come up but I suspect you are
> thinking of master map entries which are different. There are no lookups
> for the master map and so there is no opportunity to check if the map
> has changed. To see changes to the master map a re-load is needed.
>
>   
>> I know multimount entries are difficult, and are handled as one.
>>
>> But how about ldap? I assume you're talking about how maps are handled
>> here in general, but is this different/the same with ldap??
>>     
>
> As far a the master map goes there isn't any real difference which
> source is used.
>
> As far as map entries go LDAP is probably the worst since it has no
> standard, consistent information about when it was last updated
> regardless of whether those entries are the master map or map entries
> themselves.
>
> Ian
>
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs
>   

Re: ldap and reloading

[Ian Kent]

Ondrej Valousek wrote:
> Talking about ldap maps reloading - maybe it is worth mentioning that if 
> you configure automounter using the recently added DNS SRV support, 
> automounter reloads itself as per zone TTL in DNS (that includes the 
> master map, too) - right Ian?
> Ondrej
> 
> Ian Kent wrote:
>> Stef Bon wrote:
>>   
>>> Ian Kent wrote:
>>>     
>>>> Stef Bon wrote:
>>>>  
>>>>       
>>>>> Hello,
>>>>>
>>>>> when using static file maps, or even executable maps, when the map
>>>>> changes, you'll have to reload the daemon to make these changes
>>>>> effective.
>>>>>
>>>>> How does this work when all the data is in ldap? Does the automounter
>>>>> still creates a sort of snapshot of the map in memory, and to reread
>>>>> the data provided by ldap, a reload is neccesary? ldap is well know
>>>>> for that this is not needed. For example with postfix, it can use
>>>>> static maps, and when something changes, it has te be restarted. But
>>>>> when the lookup data (for local users for example) is in ldap, this
>>>>> is not neccesary.
>>>>>     
>>>>>         
>>>> That's not quite right.
>>>>
>>>> If you use the "browse" option then the entire map must be read in at
>>>> start. If not then autofs remembers entries that it has seen and
>>>> attempts to check their currency at lookup.
>>>>
>>>> Each lookup should check if the entry is still up to date and attempts
>>>> to work out if the map has changed (although it's not quite as simple as
>>>> that). If we think the map has changed a re-load should be triggered
>>>> internally. Following the (or any) re-load there is a cleanup which is
>>>> probably why it looks like map changes aren't seen. Any changes in
>>>> multi-mount entries cannot be seen until after they have expired away,
>>>> because of the need to maintain the context of the entry over the
>>>> duration of the mount.
>>>>
>>>> Direct maps don't quite do this properly, partly because of the way they
>>>> work and partly because of an issue I haven't addressed yet.
>>>>
>>>> Clearly, with program maps, we need to rely on the re-load to a large
>>>> extent but a best effort is made to work out if the entry is stale,
>>>> however, we just don't have anything really to use to establish this, so
>>>>  a re-load is needed to clean them up.
>>>>   
>>>>       
>>> Ok Ian,
>>>
>>> maybe you're right. You probably are, I cannot discuss this with you,
>>> you're the expert.
>>>
>>> You say that it may look like map changes aren't seen. How, as developer
>>> of a construction making use of autofs,
>>> can I see/check it's reloaded. Are there any indications, or triggers
>>> which always lead to this internal reloading?
>>> BTW I also use reloading when a map/mountpoint is added or removed, so
>>> not only when a map changes.

Not quite.

On re-load, if the last time we got the list of servers is greater than
the ttl it will lookup there server list again before attempting to read
from them. Map entry lookups them selves don't check this ttl so a
re-load is only done if we think the map has changed.

>>>     
>> OK, but I think we're not talking about the same thing.
>>
>> I always think about map entries when this come up but I suspect you are
>> thinking of master map entries which are different. There are no lookups
>> for the master map and so there is no opportunity to check if the map
>> has changed. To see changes to the master map a re-load is needed.
>>
>>   
>>> I know multimount entries are difficult, and are handled as one.
>>>
>>> But how about ldap? I assume you're talking about how maps are handled
>>> here in general, but is this different/the same with ldap??
>>>     
>> As far a the master map goes there isn't any real difference which
>> source is used.
>>
>> As far as map entries go LDAP is probably the worst since it has no
>> standard, consistent information about when it was last updated
>> regardless of whether those entries are the master map or map entries
>> themselves.
>>
>> Ian
>>
>> _______________________________________________
>> autofs mailing list
>> autofs@linux.kernel.org
>> http://linux.kernel.org/mailman/listinfo/autofs
>>   
> 
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs

Re: ldap and reloading

[Stef Bon]

Ian Kent wrote:
> Ondrej Valousek wrote:
>   
>> Talking about ldap maps reloading - maybe it is worth mentioning that if 
>> you configure automounter using the recently added DNS SRV support, 
>> automounter reloads itself as per zone TTL in DNS (that includes the 
>> master map, too) - right Ian?
>> Ondrej
>>
>>     
> Not quite.
>
> On re-load, if the last time we got the list of servers is greater than
> the ttl it will lookup there server list again before attempting to read
> from them. Map entry lookups them selves don't check this ttl so a
> re-load is only done if we think the map has changed.
>   

Well, this is all getting very complicated. Too complicated for me.

The only thing I'm using autofs for is creating a autofs managed 
directory in the homedirectory, for mounting local (USB) and remote (SMB 
shares) resources. To make this work, the autofs daemon has to reread 
the configuration of maps multiple times.

It would be so much easier if there is a sort of autofsctl utility for 
adding/removing mountpoints, and changing maps.

for example:

a. add a mounpoint and map.
b. delete a mountpoint

c. change a map.

Now when something changes, the global remounter gets a reload signal, 
which is still buggy. I'm still having problems here, but since I've 
upgraded the kernel to linux-2.6.29.4 I've got less problems.

Stef

Re: ldap and reloading

[Ian Kent]

Stef Bon wrote:
> Ian Kent wrote:
>> Ondrej Valousek wrote:
>>  
>>> Talking about ldap maps reloading - maybe it is worth mentioning that
>>> if you configure automounter using the recently added DNS SRV
>>> support, automounter reloads itself as per zone TTL in DNS (that
>>> includes the master map, too) - right Ian?
>>> Ondrej
>>>
>>>     
>> Not quite.
>>
>> On re-load, if the last time we got the list of servers is greater than
>> the ttl it will lookup there server list again before attempting to read
>> from them. Map entry lookups them selves don't check this ttl so a
>> re-load is only done if we think the map has changed.
>>   
> 
> Well, this is all getting very complicated. Too complicated for me.
> 
> The only thing I'm using autofs for is creating a autofs managed
> directory in the homedirectory, for mounting local (USB) and remote (SMB
> shares) resources. To make this work, the autofs daemon has to reread
> the configuration of maps multiple times.
> 
> It would be so much easier if there is a sort of autofsctl utility for
> adding/removing mountpoints, and changing maps.
> 
> for example:
> 
> a. add a mounpoint and map.
> b. delete a mountpoint
> 
> c. change a map.

Yes, that's a good idea and we've thought about doing a control program
several times in the past but, it's quite a bit of work and is well
outside the scope for the 5.0 series.

> 
> Now when something changes, the global remounter gets a reload signal,
> which is still buggy. I'm still having problems here, but since I've
> upgraded the kernel to linux-2.6.29.4 I've got less problems.

And we need to fix the reloading problem anyway.
I've had a brief look around regarding your report and hope to get back
to it and spend more time on it soon, sorry.

Ian