udp ports not being released

udp ports not being released

[Dwight Marzolf]

We are running RedHat 8, kernel 2.4.20-20.8smp and autofs 4.1.2.  We 
were having a variety of automount problems until a kernel patch was 
recommended.  We added this patch, recompiled our kernel, and 
distributed the patched kernel to all of our workstations.  About 90% of 
our problems disappeared.  But one problem seems to remain.

When /net is used to mount directories, most of the time the udp port 
used for that nfs mount operation is not released once the /net mount 
times out.   Depending on the level of activity on a machine, over the 
period of several days, all the available reserved ports are used up.  
At that point the user on the workstation cannot cd to any mount points 
in our automount maps except the ones already mounted.  In the 
/var/log/messages the following error occurs for attempted mounts:

RPC: Can't bind to reserved port

In my experiments I found that doing a "umount -l /net" releases all the 
udp ports immediately.  Our workaround has been to kill all the autofs 
processes, do a umount -l on all mount points in /etc/mtab, and them 
restart the autofs process.

My question is (1) is there any fix for this problem or some switch I 
need to turn on/off in order for these udp ports to be released when the 
mount point times out and (2) is there some command that I can use to 
release these ports rather than going through the multiple step process 
I described above to clean up a machine?

Dwight Marzolf




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: udp ports not being released

[Mike Waychison]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dwight Marzolf wrote:
> We are running RedHat 8, kernel 2.4.20-20.8smp and autofs 4.1.2.  We
> were having a variety of automount problems until a kernel patch was
> recommended.  We added this patch, recompiled our kernel, and
> distributed the patched kernel to all of our workstations.  About 90% of
> our problems disappeared.  But one problem seems to remain.
> 
> When /net is used to mount directories, most of the time the udp port
> used for that nfs mount operation is not released once the /net mount
> times out.   Depending on the level of activity on a machine, over the
> period of several days, all the available reserved ports are used up. 
> At that point the user on the workstation cannot cd to any mount points
> in our automount maps except the ones already mounted.  In the
> /var/log/messages the following error occurs for attempted mounts:

IIRC, automount 4.1.2 had a fd leak, that was fixed in 4.1.3

Ian?

> 
> RPC: Can't bind to reserved port
> 
> In my experiments I found that doing a "umount -l /net" releases all the
> udp ports immediately.  Our workaround has been to kill all the autofs
> processes, do a umount -l on all mount points in /etc/mtab, and them
> restart the autofs process.
> 
> My question is (1) is there any fix for this problem or some switch I
> need to turn on/off in order for these udp ports to be released when the
> mount point times out and (2) is there some command that I can use to
> release these ports rather than going through the multiple step process
> I described above to clean up a machine?
> 
> Dwight Marzolf
> 
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs


- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE:  The opinions expressed in this email are held by me,
and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBokRTdQs4kOxk3/MRAmNFAJ9vYo0GjHbM2RlquSgAVjbJ32lnlQCeOZkc
8hRRC7gWmZwspf9ySrnkZE4=
=H26H
-----END PGP SIGNATURE-----


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: udp ports not being released

[Ian Kent]

On Mon, 22 Nov 2004, Mike Waychison wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Dwight Marzolf wrote:
> > We are running RedHat 8, kernel 2.4.20-20.8smp and autofs 4.1.2.  We
> > were having a variety of automount problems until a kernel patch was
> > recommended.  We added this patch, recompiled our kernel, and
> > distributed the patched kernel to all of our workstations.  About 90% of
> > our problems disappeared.  But one problem seems to remain.
> > 
> > When /net is used to mount directories, most of the time the udp port
> > used for that nfs mount operation is not released once the /net mount
> > times out.   Depending on the level of activity on a machine, over the
> > period of several days, all the available reserved ports are used up. 
> > At that point the user on the workstation cannot cd to any mount points
> > in our automount maps except the ones already mounted.  In the
> > /var/log/messages the following error occurs for attempted mounts:
> 
> IIRC, automount 4.1.2 had a fd leak, that was fixed in 4.1.3

Yes and also in 4.1.3. There is a patch on kernel.org but it has been 
superceeded and not updated. I'll update it soon as I can (called 
autofs-4.1.3-sock-leak-fix.patch).

I'm not sure about the socket leaks anyway. We fix them because it's good 
practice but it appears that most of this happens in sub-processes, which 
once finished close all their open fds. This certainly seems to work 
correctly from what I've tested.

For Redhat Jeff Moyers' rpms track the important fixes. Dwight, you should 
try using one of them since you have problems.

> 
> Ian?
> 
> > 
> > RPC: Can't bind to reserved port
> > 
> > In my experiments I found that doing a "umount -l /net" releases all the
> > udp ports immediately.  Our workaround has been to kill all the autofs
> > processes, do a umount -l on all mount points in /etc/mtab, and them
> > restart the autofs process.
> > 
> > My question is (1) is there any fix for this problem or some switch I
> > need to turn on/off in order for these udp ports to be released when the
> > mount point times out and (2) is there some command that I can use to
> > release these ports rather than going through the multiple step process
> > I described above to clean up a machine?

See above.

There is of course the possiblity this is a new problem so let us know 
about your progress if any.

Ian



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: [NFS] udp ports not being released

[Jeff Moyer]

==> Regarding [autofs] Re: [NFS] udp ports not being released; Mike Waychison <Michael.Waychison@sun.com> adds:

Michael.Waychison> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Michael.Waychison> Dwight Marzolf wrote:
>> We are running RedHat 8, kernel 2.4.20-20.8smp and autofs 4.1.2.  We
>> were having a variety of automount problems until a kernel patch was
>> recommended.  We added this patch, recompiled our kernel, and
>> distributed the patched kernel to all of our workstations.  About 90% of
>> our problems disappeared.  But one problem seems to remain.
>> 
>> When /net is used to mount directories, most of the time the udp port
>> used for that nfs mount operation is not released once the /net mount
>> times out.  Depending on the level of activity on a machine, over the
>> period of several days, all the available reserved ports are used up. At
>> that point the user on the workstation cannot cd to any mount points in
>> our automount maps except the ones already mounted.  In the
>> /var/log/messages the following error occurs for attempted mounts:

Michael.Waychison> IIRC, automount 4.1.2 had a fd leak, that was fixed in
Michael.Waychison> 4.1.3

Michael.Waychison> Ian?

I'm not Ian, but I can answer.  That leak still exists in 4.1.3.  There are
a couple of patches posted that deal with this problem and a related
problem.  One fixes the socket leak, the other teaches the udp rpc_ping
function to not use reserved ports.

-Jeff