autofs illustrations

autofs illustrations

[wengang wang]

Hi all,

Currently, I am working on debugging for autofs/automount. But I am not 
very clear about how autofs/automount works.
Is there any illustration on autofs/automount?

your answer will be greatly appreciated!

best regards,
wengang.

Re: autofs illustrations

[Jeff Moyer]

==> Regarding [autofs] autofs illustrations; wengang wang <wen.gang.wang@oracle.com> adds:

wen.gang.wang> Hi all,

wen.gang.wang> Currently, I am working on debugging for
wen.gang.wang> autofs/automount. But I am not very clear about how
wen.gang.wang> autofs/automount works.  Is there any illustration on
wen.gang.wang> autofs/automount?

wen.gang.wang> your answer will be greatly appreciated!

http://people.redhat.com/jmoyer/autofs_linux_kongress.pdf

-Jeff

Re: autofs illustrations

[Fletcher Mattox]

Jeff Moyer writes:

> http://people.redhat.com/jmoyer/autofs_linux_kongress.pdf

Jeff,

Nice paper.  Thanks!  I'm suffering from RPC port exhaustion (you
responded to one of my queries last week), so I am especially interested
in your section on port allocation.  You state

	When a service requests an RPC connection, binding to
	a reserved port is the default.  The RPC layer scans
	ports starting from 800 down until it finds one that
	is unallocated.

My experience is that the 2.6.17.4 kernel allocates between 512 and 1024
and then begins with unreserved ports >33000.  Yeah, I know.  It's just
a nit, but I thought I'd tell you anyway.

The most interesting part of the paper was the comment about a practical
limit of 100 mounts in rapid succession.  I am certain this is the limit
we have bumped into.  Do you see any improvement in this in autofs 5?

Unprivileged ports and/or UDP are not viable options for us, so I am
forced to increase the timeout from 5 minutes to 24 hours, which in
practice means they are always mounted.  We have about 400 automounted
filesystems, so the only long term solution for us is to try to coalesce
them to less than 100.  Very painful.

Thanks again,
Fletcher

Re: autofs illustrations

[Peter Staubach]

Fletcher Mattox wrote:
> Jeff Moyer writes:
>
>   
>> http://people.redhat.com/jmoyer/autofs_linux_kongress.pdf
>>     
>
> Jeff,
>
> Nice paper.  Thanks!  I'm suffering from RPC port exhaustion (you
> responded to one of my queries last week), so I am especially interested
> in your section on port allocation.  You state
>
> 	When a service requests an RPC connection, binding to
> 	a reserved port is the default.  The RPC layer scans
> 	ports starting from 800 down until it finds one that
> 	is unallocated.
>
> My experience is that the 2.6.17.4 kernel allocates between 512 and 1024
> and then begins with unreserved ports >33000.  Yeah, I know.  It's just
> a nit, but I thought I'd tell you anyway.
>
> The most interesting part of the paper was the comment about a practical
> limit of 100 mounts in rapid succession.  I am certain this is the limit
> we have bumped into.  Do you see any improvement in this in autofs 5?
>
> Unprivileged ports and/or UDP are not viable options for us, so I am
> forced to increase the timeout from 5 minutes to 24 hours, which in
> practice means they are always mounted.  We have about 400 automounted
> filesystems, so the only long term solution for us is to try to coalesce
> them to less than 100.  Very painful.

You have 400 automounted file systems, all of which need to be mounted at
the same time?  If so, I might suggest that static mounts might better serve
your needs.  Or, rethink the application and deployment.

    Thanx...

       ps

Re: autofs illustrations

[Fletcher Mattox]

> Fletcher Mattox wrote:
> ...
> > Unprivileged ports and/or UDP are not viable options for us, so I am
> > forced to increase the timeout from 5 minutes to 24 hours, which in
> > practice means they are always mounted.  We have about 400 automounted
> > filesystems, so the only long term solution for us is to try to coalesce
> > them to less than 100.  Very painful.
> 
> You have 400 automounted file systems, all of which need to be mounted at
> the same time?  

Not all of them, but certainly more than 100 of them, which seems to
be the limit we are talking about. 

> If so, I might suggest that static mounts might better serve
> your needs.  

Really?  I think this is the first time I have ever heard someone
advocate static mounts as a solution to a large number of filesystems
(especially ones that tend to appear and disappear frequently).
But you are right, and this is effectively the solution we arrived at
by increasing the timeout.

> Or, rethink the application and deployment.

I think that's what said in my penultimate sentence. :)

Thanks
Fletcher

Re: autofs illustrations

[Peter Staubach]

Fletcher Mattox wrote:
>> Fletcher Mattox wrote:
>> ...
>>     
>>> Unprivileged ports and/or UDP are not viable options for us, so I am
>>> forced to increase the timeout from 5 minutes to 24 hours, which in
>>> practice means they are always mounted.  We have about 400 automounted
>>> filesystems, so the only long term solution for us is to try to coalesce
>>> them to less than 100.  Very painful.
>>>       
>> You have 400 automounted file systems, all of which need to be mounted at
>> the same time?  
>>     
>
> Not all of them, but certainly more than 100 of them, which seems to
> be the limit we are talking about. 
>
>   

Hmm.  Perhaps Chuck Lever's work to reduce the number of required TCP
connections would help in this sort of deployment.

>> If so, I might suggest that static mounts might better serve
>> your needs.  
>>     
>
> Really?  I think this is the first time I have ever heard someone
> advocate static mounts as a solution to a large number of filesystems
> (especially ones that tend to appear and disappear frequently).
> But you are right, and this is effectively the solution we arrived at
> by increasing the timeout.
>
>   

Well, if the file systems tend to get mounted and then stay mounted,
then automounting does little good and can be harmful.

Automounting also does not tend to work well with large numbers of
file systems and not always for the scalability reasons which are
being seen here.
>> Or, rethink the application and deployment.
>>     
>
> I think that's what said in my penultimate sentence. :)

Yup...  :-)

       ps

Re: autofs illustrations

[Jeff Moyer]

Peter Staubach <staubach@redhat.com> writes:

> Fletcher Mattox wrote:
>>> Fletcher Mattox wrote:
>>> ...
>>>     
>>>> Unprivileged ports and/or UDP are not viable options for us, so I am
>>>> forced to increase the timeout from 5 minutes to 24 hours, which in
>>>> practice means they are always mounted.  We have about 400 automounted
>>>> filesystems, so the only long term solution for us is to try to coalesce
>>>> them to less than 100.  Very painful.
>>>>       
>>> You have 400 automounted file systems, all of which need to be mounted at
>>> the same time?  
>>>     
>>
>> Not all of them, but certainly more than 100 of them, which seems to
>> be the limit we are talking about. 
>>
>>   
>
> Hmm.  Perhaps Chuck Lever's work to reduce the number of required TCP
> connections would help in this sort of deployment.
>
>>> If so, I might suggest that static mounts might better serve
>>> your needs.  
>>>     
>>
>> Really?  I think this is the first time I have ever heard someone
>> advocate static mounts as a solution to a large number of filesystems
>> (especially ones that tend to appear and disappear frequently).
>> But you are right, and this is effectively the solution we arrived at
>> by increasing the timeout.
>>
>>   
>
> Well, if the file systems tend to get mounted and then stay mounted,
> then automounting does little good and can be harmful.

Yes, but at least you get the advantage of central management of your
mount points.

> Automounting also does not tend to work well with large numbers of
> file systems and not always for the scalability reasons which are
> being seen here.

And yet people deploy things this way, and it works.

-jeff

Re: autofs illustrations

[Michael Blandford]

Peter Staubach wrote:

>Fletcher Mattox wrote:
>  
>
>>Jeff Moyer writes:
>>
>>
>>Unprivileged ports and/or UDP are not viable options for us, so I am
>>forced to increase the timeout from 5 minutes to 24 hours, which in
>>practice means they are always mounted.  We have about 400 automounted
>>filesystems, so the only long term solution for us is to try to coalesce
>>them to less than 100.  Very painful.
>>    
>>
>
>You have 400 automounted file systems, all of which need to be mounted at
>the same time?  If so, I might suggest that static mounts might better serve
>your needs.  Or, rethink the application and deployment.
>
>  
>


There are definitely two cases here.

1) There are going to be environments that are set up without much 
thought of the future.  You may have 400 mounts that you need for  a 
project all at one time because of small filesystems, inefficient 
layout, etc.  There may be only 400 mounts on the file server or servers.

2) There are projects/environments that have several thousands ( > 10k ) 
of filesystems that are layed out efficiently but still require more 
than > 1000 mounts at any given moment due to the large data requirements.

Either way, we need autofs and the NFS subsystem be able to handle 
this.  Many of the other UNIX vendors seems to handle this situation well.

Some Linux vendors already include patches to multiplex rpc to allow > 
1000 mounts.  However, the issue of quickly mounting hundreds/thousands 
of mounts is still a challenge - especially with NFS over TCP.

As a temporary solution, we have added a new option to the mount command 
to allow it to use udp for the handshaking and then only use tcp for the 
final mount.  This allows us to handle many more simultaneous mounts but 
it definitely a hack and won't work for people who can only use tcp.


Michael

Re: autofs illustrations

[Ian Kent]

On Tue, 2006-11-07 at 11:19 -0600, Fletcher Mattox wrote:
> Jeff Moyer writes:
> 
> > http://people.redhat.com/jmoyer/autofs_linux_kongress.pdf
> 
> Jeff,
> 
> Nice paper.  Thanks!  I'm suffering from RPC port exhaustion (you
> responded to one of my queries last week), so I am especially interested
> in your section on port allocation.  You state
> 
> 	When a service requests an RPC connection, binding to
> 	a reserved port is the default.  The RPC layer scans
> 	ports starting from 800 down until it finds one that
> 	is unallocated.
> 
> My experience is that the 2.6.17.4 kernel allocates between 512 and 1024
> and then begins with unreserved ports >33000.  Yeah, I know.  It's just
> a nit, but I thought I'd tell you anyway.

We know there have been changes in that area but I don't think they were
in place at the time this was written. However, the changes were in
place when the paper was updated and that section wasn't updated.

Bit naughty really.

The updated version is in the 2006 OLS publication.

> 
> The most interesting part of the paper was the comment about a practical
> limit of 100 mounts in rapid succession.  I am certain this is the limit
> we have bumped into.  Do you see any improvement in this in autofs 5?

Not really, about 3/4 of the additional port opens are done in mount,
not autofs.

> 
> Unprivileged ports and/or UDP are not viable options for us, so I am
> forced to increase the timeout from 5 minutes to 24 hours, which in
> practice means they are always mounted.  We have about 400 automounted
> filesystems, so the only long term solution for us is to try to coalesce
> them to less than 100.  Very painful.

Understood. We had similar constraints at my last employer.

I'm not aware of any work being actively done on this.
Nobody that cares has any time at the moment.

Ian

Re: autofs illustrations

[Peter Staubach]

Ian Kent wrote:
>
>   
>> Unprivileged ports and/or UDP are not viable options for us, so I am
>> forced to increase the timeout from 5 minutes to 24 hours, which in
>> practice means they are always mounted.  We have about 400 automounted
>> filesystems, so the only long term solution for us is to try to coalesce
>> them to less than 100.  Very painful.
>>     
>
> Understood. We had similar constraints at my last employer.
>
> I'm not aware of any work being actively done on this.
> Nobody that cares has any time at the moment.
>
>   

Actually, there is work on going to reduce the number of connections 
from the
client to the server from one per mount to one per client server 
combination.
Solaris has always done this and it works out quite well in practice.

       ps

Re: autofs illustrations

[Ian Kent]

On Wed, 2006-11-08 at 08:25 -0500, Peter Staubach wrote:
> Ian Kent wrote:
> >
> >   
> >> Unprivileged ports and/or UDP are not viable options for us, so I am
> >> forced to increase the timeout from 5 minutes to 24 hours, which in
> >> practice means they are always mounted.  We have about 400 automounted
> >> filesystems, so the only long term solution for us is to try to coalesce
> >> them to less than 100.  Very painful.
> >>     
> >
> > Understood. We had similar constraints at my last employer.
> >
> > I'm not aware of any work being actively done on this.
> > Nobody that cares has any time at the moment.
> >
> >   
> 
> Actually, there is work on going to reduce the number of connections 
> from the
> client to the server from one per mount to one per client server 
> combination.
> Solaris has always done this and it works out quite well in practice.

Excellent.
Tell me more?

Ian

Re: autofs illustrations

[Joe Pruett]

> > Actually, there is work on going to reduce the number of connections 
> > from the
> > client to the server from one per mount to one per client server 
> > combination.
> > Solaris has always done this and it works out quite well in practice.
> 
> Excellent.
> Tell me more?

this sure sounds like the old solaris automount setup of:

foo	server:/path/disk:foo
bar	server:/path/disk:bar

hitting foo would mount server:/path/disk and then symlink/bind for foo.  
hitting bar within the timeout period would just cause another 
symlink/bind instead of another mount.

i am mimicking this on my setup with two maps.  one for disks and then one 
for home dirs that points to the disks mount.  it still hiccups 
occasionally since the two automounts don't really know about each other.  
having this functionality back in automount would be great.

Re: autofs illustrations

[Jeff Moyer]

==> On Wed, 8 Nov 2006 14:34:27 -0800 (PST), Joe Pruett <joey@clean.q7.com> said:

Joe> > > Actually, there is work on going to reduce the number of
Joe> connections > > from the > > client to the server from one per
Joe> mount to one per client server > > combination.  > > Solaris has
Joe> always done this and it works out quite well in practice.
Joe> > 
Joe> > Excellent.  > Tell me more?

Joe> this sure sounds like the old solaris automount setup of:

Joe> foo server:/path/disk:foo bar server:/path/disk:bar

Joe> hitting foo would mount server:/path/disk and then symlink/bind
Joe> for foo.  hitting bar within the timeout period would just cause
Joe> another symlink/bind instead of another mount.

Quite different.  I think Pete is referring to connection sharing.

-Jeff

Re: autofs illustrations

[Peter Staubach]

Jeff Moyer wrote:
> ==> On Wed, 8 Nov 2006 14:34:27 -0800 (PST), Joe Pruett <joey@clean.q7.com> said:
>
> Joe> > > Actually, there is work on going to reduce the number of
> Joe> connections > > from the > > client to the server from one per
> Joe> mount to one per client server > > combination.  > > Solaris has
> Joe> always done this and it works out quite well in practice.
> Joe> > 
> Joe> > Excellent.  > Tell me more?
>
> Joe> this sure sounds like the old solaris automount setup of:
>
> Joe> foo server:/path/disk:foo bar server:/path/disk:bar
>
> Joe> hitting foo would mount server:/path/disk and then symlink/bind
> Joe> for foo.  hitting bar within the timeout period would just cause
> Joe> another symlink/bind instead of another mount.
>
> Quite different.  I think Pete is referring to connection sharing.

Quite different indeed.  I am talking about managing connections
separately from mounts.  There isn't any logical reason that these
two semantics need to be tied to each other.

       ps