RE: autofs scalability

RE: autofs scalability

[Ogden, Aaron A.]

> -----Original Message-----
> From: autofs-bounces@linux.kernel.org 
> [mailto:autofs-bounces@linux.kernel.org] On Behalf Of Ian Kent
> Sent: Tuesday, December 02, 2003 7:49 AM
> To: Chris Croswhite
> Cc: autofs@linux.kernel.org
> Subject: RE: [autofs] making the mountpoints visible all the 
> time (fwd)
> 
> Hi guys,
> 
> I have some 2.6 patches.
> They can be found at 
> http://www.kernel.org/pub/linux/kernel/people/raven/autofs4-2.6
> Apply them in number order please.
> 
> Please leave out the max_anon_2.patch initially. We need to establish
what 
> the status of the kernel module is before trying this patch. It was 
> written by Mike Waychison to increase the number of anonymous devices.

> Once the status patches is established it would be good to how the 
> max_anon patch goes with over 255 mounts and less than about 790 (rpc 
> doesn't allow more than that at the moment).
> 
... <snip>

Hello all,
I was reading through Joe Pranevich's "Wonderful World of Linux 2.6" and
I came across this, I thought you might find it interesting. (source:
http://kniggit.net/wwol26.html)  The "Network Filesystems" section makes
for interesting reading as well.  If I am reading this correctly, the
'unnamed device' limit has been raised from 256 to 4096 in kernel 2.6,
can anyone confirm this?  If this is the case the only barrier left to
remove is the RPC port limitation of ~800 open ports.  If the
connections were multiplexed such that all connections to a given server
used the same port then 800 ports would stretch a lot farther than they
do now...

"Another major scalability improvement in Linux 2.6 is that the kernel
itself can now not only support more types of devices, but also support
more devices of a single type. Under all iterations of Linux (and
indeed, most UNIX-derived operating systems), users and applications
running on a system communicate with the attached hardware using
numbered device nodes. (The entries in the "/dev" directory.) These
device nodes were limited to 255 "major" devices (generally, one type of
device gets one or more device nodes) and 255 "minor" numbers
(generally, specific devices of that type.) For example, the "/dev/sda2"
device (the second partition on the first detected SCSI disk), gets a
major number of 8, common for all SCSI devices, and a minor number of 2
to indicate the second partition. Different device types allocate their
major and minor numbers differently, so it can't be said with assurance
how many devices you can have on a Linux system. Unfortunately, this
system breaks down badly on large systems where it would be possible,
for example, to have many more than 255 of any specific device in a
system. (Think large storage arrays, print farms, etc.) Under Linux 2.6,
these limitations have been eased to allow for 4095 major device types
and a more than a million subdevices per type. This increase should be
more than adequate to support high-end systems for the time being."

Re: RE: autofs scalability

[Tim Hockin]

On Wed, Dec 03, 2003 at 01:19:27PM -0600, Ogden, Aaron A. wrote:
> 'unnamed device' limit has been raised from 256 to 4096 in kernel 2.6,
> can anyone confirm this?  If this is the case the only barrier left to
> remove is the RPC port limitation of ~800 open ports.  If the
> connections were multiplexed such that all connections to a given server
> used the same port then 800 ports would stretch a lot farther than they
> do now...

We're investigating the 800 limit reasoning.  2.6.(last I looked) has not
removed the 256 limit, but has made it so that it CAN BE removed.  We hav a
prelim patch for that, which I wan tot rework, some, maybe.

Re: RE: autofs scalability

[Mike Waychison]

[-- Attachment #1.1: Type: text/plain, Size: 1403 bytes --]

Ogden, Aaron A. wrote:

>Hello all,
>I was reading through Joe Pranevich's "Wonderful World of Linux 2.6" and
>I came across this, I thought you might find it interesting. (source:
>http://kniggit.net/wwol26.html)  The "Network Filesystems" section makes
>for interesting reading as well.  If I am reading this correctly, the
>'unnamed device' limit has been raised from 256 to 4096 in kernel 2.6,
>can anyone confirm this?  
>

No.  The major count went from 256 -> 4096.  The minor count went from 
256 to 1048576.  The maximum pseudo-block-device limit is still 256 
currently.

>If this is the case the only barrier left to
>remove is the RPC port limitation of ~800 open ports.  If the
>connections were multiplexed such that all connections to a given server
>used the same port then 800 ports would stretch a lot farther than they
>do now...
>
>  
>
Yes.  This has been identified as an upper limit.  :(

But you're right, these two limits are blocking being able to mount 
large numbers of NFS mounts.


-- 
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice
mailto: Michael.Waychison@Sun.COM
http://www.sun.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE:  The opinions expressed in this email are held by me, 
and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 


[-- Attachment #1.2: Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

RE: autofs scalability

[Ian Kent]

Hi Aaron,

Long time no hear.

Good to hear from you.

On Wed, 3 Dec 2003, Ogden, Aaron A. wrote:

>
> Hello all,
> I was reading through Joe Pranevich's "Wonderful World of Linux 2.6" and
> I came across this, I thought you might find it interesting. (source:
> http://kniggit.net/wwol26.html)  The "Network Filesystems" section makes
> for interesting reading as well.  If I am reading this correctly, the
> 'unnamed device' limit has been raised from 256 to 4096 in kernel 2.6,
> can anyone confirm this?  If this is the case the only barrier left to
> remove is the RPC port limitation of ~800 open ports.  If the
> connections were multiplexed such that all connections to a given server
> used the same port then 800 ports would stretch a lot farther than they
> do now...

Sorry to say that the code that limits the number of anonymous mounts
to 255 does in fact remain. At least it was last time I checked.

I have been studying the rpc/nfs code but it is fairly complex so I am
not able to work on either of the two thinks you mention yet, because of
it.

-- 

   ,-._|\    Ian Kent
  /      \   Perth, Western Australia
  *_.--._/   E-mail: raven@themaw.net
        v    Web: http://themaw.net/

Re: RE: autofs scalability

[Ian Kent]

On Wed, 3 Dec 2003, Tim Hockin wrote:

> On Wed, Dec 03, 2003 at 01:19:27PM -0600, Ogden, Aaron A. wrote:
> > 'unnamed device' limit has been raised from 256 to 4096 in kernel 2.6,
> > can anyone confirm this?  If this is the case the only barrier left to
> > remove is the RPC port limitation of ~800 open ports.  If the
> > connections were multiplexed such that all connections to a given server
> > used the same port then 800 ports would stretch a lot farther than they
> > do now...
>
> We're investigating the 800 limit reasoning.  2.6.(last I looked) has not
> removed the 256 limit, but has made it so that it CAN BE removed.  We hav a
> prelim patch for that, which I wan tot rework, some, maybe.
>

Are you refering to the sysctl recommendation.

I would like to do it but haven't had time.

I'll let you know if I get to start work on it.


-- 

   ,-._|\    Ian Kent
  /      \   Perth, Western Australia
  *_.--._/   E-mail: raven@themaw.net
        v    Web: http://themaw.net/

Re: RE: autofs scalability

[Tim Hockin]

On Thu, Dec 04, 2003 at 09:37:44AM +0800, Ian Kent wrote:
> > removed the 256 limit, but has made it so that it CAN BE removed.  We hav a
> > prelim patch for that, which I wan tot rework, some, maybe.
> >
> 
> Are you refering to the sysctl recommendation.
> 
> I would like to do it but haven't had time.
> 
> I'll let you know if I get to start work on it.

Ditto. It's simple, but it just hasn't been high enough priority.

Re: RE: autofs scalability

[H. Peter Anvin]

Tim Hockin wrote:
> On Thu, Dec 04, 2003 at 09:37:44AM +0800, Ian Kent wrote:
> 
>>>removed the 256 limit, but has made it so that it CAN BE removed.  We hav a
>>>prelim patch for that, which I wan tot rework, some, maybe.
>>>
>>
>>Are you refering to the sysctl recommendation.
>>
>>I would like to do it but haven't had time.
>>
>>I'll let you know if I get to start work on it.
> 
> 
> Ditto. It's simple, but it just hasn't been high enough priority.
> 

It really would be better if it was dynamic.  Any fixed limit (other
than 2^20 minors) is really  a hack.

	-hpa

Re: RE: autofs scalability

[Tim Hockin]

On Thu, Dec 04, 2003 at 12:04:55PM -0800, H. Peter Anvin wrote:
> > Ditto. It's simple, but it just hasn't been high enough priority.
> > 
> 
> It really would be better if it was dynamic.  Any fixed limit (other
> than 2^20 minors) is really  a hack.

You either allocate 2^20 bits for every user, or you allocate it as needed
(and handle alloc failures when they are least expected) or you sysctl it
and let the admin decide early on whether she wants to allow large numbers
of pseudo-devs.


-- 
Tim Hockin
Sun Microsystems, Linux Software Engineering
thockin@sun.com
All opinions are my own, not Sun's

Re: RE: autofs scalability

[H. Peter Anvin]

Tim Hockin wrote:
> 
> You either allocate 2^20 bits for every user, or you allocate it as needed
> (and handle alloc failures when they are least expected) or you sysctl it
> and let the admin decide early on whether she wants to allow large numbers
> of pseudo-devs.
> 

I can't imagine that these are allocated left and center... it should
pretty much be mount and nothing else.  So yes, it's some more
complexity but not really significant.

2^20 bits = 128K, which falls in the "it's a bit on the expensive side,
but doable" class.

Fixed limits suck.  And yes, I'm also guilty of having put them in in
the past.  They still suck.

	-hpa

Re: RE: autofs scalability

[Tim Hockin]

On Thu, Dec 04, 2003 at 03:09:15PM -0800, H. Peter Anvin wrote:
> > You either allocate 2^20 bits for every user, or you allocate it as needed
> > (and handle alloc failures when they are least expected) or you sysctl it
> > and let the admin decide early on whether she wants to allow large numbers
> > of pseudo-devs.

> I can't imagine that these are allocated left and center... it should
> pretty much be mount and nothing else.  So yes, it's some more
> complexity but not really significant.
> 
> 2^20 bits = 128K, which falls in the "it's a bit on the expensive side,
> but doable" class.
> 
> Fixed limits suck.  And yes, I'm also guilty of having put them in in
> the past.  They still suck.


I think we agree about fixed limits, I'm just wary to spend 128k of every
user's box for it, which is why I mentioned sysctl.

Tim
-- 
Tim Hockin
Sun Microsystems, Linux Software Engineering
thockin@sun.com
All opinions are my own, not Sun's