[NFS] Attempting to mount a large number of directories

[NFS] Attempting to mount a large number of directories

[Fabian Steiner]

[-- Attachment #1: Type: text/plain, Size: 1616 bytes --]

Hello!

After having finished the migration to our new setup (user home directories 
are mounted automatically using autofs [4.1.4-8ubuntu1] and NFSv3) we are 
experiencing some problems which use to occur whenever many users attempt to 
access their directories. If such an issue appears the following messages 
will be seen in /var/log/syslog of the server:

Dec 23 18:38:00 ubuntu1 mountd[8995]: refused mount request from client 
for /exports/dir157 (/exports): illegal port 59537

At first we thought it would be an autofs problem but after having tried to 
reproduce the problem employing normal nfs mounts we encountered the same 
issues. The attached scripts might help you to reproduce it.

Running these scripts reveal that mounting 105 directories work, the 106th 
fails, however, resulting in the following error: 

[...]
Working on dir 105 ...nfs bindresvport: Address already in use
OK
Working on dir 106 ...mount: 192.168.0.46:/exports/dir106: can't read 
superblock
OK
[...]

At next, I added the "insecure" option to my /etc/exports file and increased 
the value of /proc/sys/sunrpc/max_resvport from 1023 to 5000. Now 139 
directories can be mounted without any single error, then "nfs bindresvport: 
Address already in use" appears again and mounting 141-172 also works fine. 
Afterwards all the other mount attempts are failing.

Could you explain this behaviour? What does "nfs bindresvport: Address already 
in use" try to tell me? Generally speaking both the server as well as the 
client should run out of open ports.

Merry Christmas and relaxing holidays,
Fabian :-)
























[-- Attachment #2: CONSTANTS --]
[-- Type: text/plain, Size: 72 bytes --]

SERVER="192.168.0.46"
SERVER_BASE="/exports"

CLIENT_BASE="/nfs-mounts"

[-- Attachment #3: create_dirs.sh --]
[-- Type: application/x-shellscript, Size: 185 bytes --]

[-- Attachment #4: mount_all.sh --]
[-- Type: application/x-shellscript, Size: 454 bytes --]

[-- Attachment #5: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #6: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Attempting to mount a large number of directories

[Fabian Steiner]

Fabian Steiner wrote:
> Could you explain this behaviour? What does "nfs bindresvport: Address
> already in use" try to tell me? Generally speaking both the server as well
> as the client should run out of open ports.

No ideas? We really got stuck with this problem.

Fabian

Re: [NFS] Attempting to mount a large number of directories

[Ian Kent]

On Mon, 2007-12-24 at 12:00 +0100, Fabian Steiner wrote:
> Hello!
> 
> After having finished the migration to our new setup (user home directories 
> are mounted automatically using autofs [4.1.4-8ubuntu1] and NFSv3) we are 
> experiencing some problems which use to occur whenever many users attempt to 
> access their directories. If such an issue appears the following messages 
> will be seen in /var/log/syslog of the server:
> 
> Dec 23 18:38:00 ubuntu1 mountd[8995]: refused mount request from client 
> for /exports/dir157 (/exports): illegal port 59537
> 
> At first we thought it would be an autofs problem but after having tried to 
> reproduce the problem employing normal nfs mounts we encountered the same 
> issues. The attached scripts might help you to reproduce it.
> 
> Running these scripts reveal that mounting 105 directories work, the 106th 
> fails, however, resulting in the following error: 
> 
> [...]
> Working on dir 105 ...nfs bindresvport: Address already in use
> OK
> Working on dir 106 ...mount: 192.168.0.46:/exports/dir106: can't read 
> superblock
> OK
> [...]
> 
> At next, I added the "insecure" option to my /etc/exports file and increased 
> the value of /proc/sys/sunrpc/max_resvport from 1023 to 5000. Now 139 
> directories can be mounted without any single error, then "nfs bindresvport: 
> Address already in use" appears again and mounting 141-172 also works fine. 
> Afterwards all the other mount attempts are failing.
> 
> Could you explain this behaviour? What does "nfs bindresvport: Address already 
> in use" try to tell me? Generally speaking both the server as well as the 
> client should run out of open ports.

I see no-one else wanted to buy into this.
I'm not really surprised.

Judging by your attempt to use the insecure option I assume you actually
know that you're running out of port space on the client. The server, of
course, consumes only a few ports, like ports for mountd, portmap (or
rpcbind) and NFS and because it's a server and it reuses the same port
since the different port used on the client makes the tupple
clientaddr,clientport,serveraddr,serverport unique. Point is that port
space is consumed on the client not the server.

So running out of port space is a client problem and the reason it
happens is that autofs and mount probe to see if the server is up and
what version of NFS is available etc. before doing the mount. This can
result in as many a 9 ports consumed for each mount and the ports that
aren't continuing to be used can't be reused before a timeout has
passed. I think it's about 60 seconds, and is required for the TCP
implementation to function properly. So you run out of available ports
fairly quickly if you issue many mount requests at once.

Quite a bit of work has been done to reduce the port usage over time and
I think it has been included in recent nfs-utils releases so perhaps a
later nfs-utils will help.

Really, the main problem here is that socket pooling for RPC would be
really hard to do in user space, if it can be done at all.

Ian



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Attempting to mount a large number of directories

[Fabian Steiner]

Ian Kent wrote:
>> [...]
> I see no-one else wanted to buy into this.
> I'm not really surprised.

Thanks a lot for your explainations! Now it is much easier to understand how 
things work on the low-level area.

> Judging by your attempt to use the insecure option I assume you actually
> know that you're running out of port space on the client. The server, of
> course, consumes only a few ports, like ports for mountd, portmap (or
> rpcbind) and NFS and because it's a server and it reuses the same port
> since the different port used on the client makes the tupple
> clientaddr,clientport,serveraddr,serverport unique. Point is that port
> space is consumed on the client not the server.
>
> So running out of port space is a client problem and the reason it
> happens is that autofs and mount probe to see if the server is up and
> what version of NFS is available etc. before doing the mount. This can
> result in as many a 9 ports consumed for each mount and the ports that
> aren't continuing to be used can't be reused before a timeout has
> passed. I think it's about 60 seconds, and is required for the TCP
> implementation to function properly. So you run out of available ports
> fairly quickly if you issue many mount requests at once.
>
> Quite a bit of work has been done to reduce the port usage over time and
> I think it has been included in recent nfs-utils releases so perhaps a
> later nfs-utils will help.

On the affected machines nfs-utils-1.0.7 has been installed so far. Picking up 
your hint we forced an upgrade from Ubuntu Dapper to Ubuntu Gutsy with 
version 1.1.1~git-20070709 being installed and obviously this solved the 
majority number of problems. Now approx. 400 directories can be mounted. Of 
course, the entire issue requires further investigation; we are quite content 
about the current situation, though.

Fabian

Re: [NFS] Attempting to mount a large number of directories

[Ian Kent]

On Mon, 2008-01-07 at 22:28 +0100, Fabian Steiner wrote:
> Ian Kent wrote:
> >> [...]
> > I see no-one else wanted to buy into this.
> > I'm not really surprised.
> 
> Thanks a lot for your explainations! Now it is much easier to understand how 
> things work on the low-level area.
> 
> > Judging by your attempt to use the insecure option I assume you actually
> > know that you're running out of port space on the client. The server, of
> > course, consumes only a few ports, like ports for mountd, portmap (or
> > rpcbind) and NFS and because it's a server and it reuses the same port
> > since the different port used on the client makes the tupple
> > clientaddr,clientport,serveraddr,serverport unique. Point is that port
> > space is consumed on the client not the server.
> >
> > So running out of port space is a client problem and the reason it
> > happens is that autofs and mount probe to see if the server is up and
> > what version of NFS is available etc. before doing the mount. This can
> > result in as many a 9 ports consumed for each mount and the ports that
> > aren't continuing to be used can't be reused before a timeout has
> > passed. I think it's about 60 seconds, and is required for the TCP
> > implementation to function properly. So you run out of available ports
> > fairly quickly if you issue many mount requests at once.
> >
> > Quite a bit of work has been done to reduce the port usage over time and
> > I think it has been included in recent nfs-utils releases so perhaps a
> > later nfs-utils will help.
> 
> On the affected machines nfs-utils-1.0.7 has been installed so far. Picking up 
> your hint we forced an upgrade from Ubuntu Dapper to Ubuntu Gutsy with 
> version 1.1.1~git-20070709 being installed and obviously this solved the 
> majority number of problems. Now approx. 400 directories can be mounted. Of 
> course, the entire issue requires further investigation; we are quite content 
> about the current situation, though.

Great, you may also find that specifying the protocol as a mount option,
proto=tcp for example, will reduce the overhead further.

Ian