xprt_bindresvport

xprt_bindresvport

[Olaf Kirch]

Hi,

the current xprt_bindresvport implementation will search for a privileged
port by counting down from 800 to 0. I think this is a bug, because it
will potentially interfere with services trying to bind to low ports as
well. The bindresvport implementation in glibc picks from the 600-1023
range.

I also think it would be good to start at a "random" port. Otherwise,
when you reboot, the server may still have a TCB for the old connection
and send you an ACK probe when you try to connect (if all goes well), and
the client's TCP stack will RST and fail the connect. If things go
not-so-well you have a packet filter somewhere inbetween that eats the
ACK probe because its connection tracking engine thinks the connection
is in half-open and shouldn't see any SYN-less ACKs yet.

Olaf
-- 
Olaf Kirch     | Things that make Monday morning interesting, #2:
okir@suse.de   |        "We have 8,000 NFS mount points, why do we keep
---------------+ 	 running out of privileged ports?"


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: xprt_bindresvport

[Lever, Charles]

> the current xprt_bindresvport implementation will search for=20
> a privileged
> port by counting down from 800 to 0. I think this is a bug, because it
> will potentially interfere with services trying to bind to=20
> low ports as
> well.

is this idle speculation, or do you actually have a test case that
fails?  :^)

> The bindresvport implementation in glibc picks from the 600-1023
> range.

we should review what other RPC implementations do (namely the reference
implementation, Solaris).

but also notice this cuts the usable port range in half (from ~800 to
~420).  we need some form of mitigation to ensure we aren't limiting the
number of NFS mounts a client can have.

> I also think it would be good to start at a "random" port. Otherwise,
> when you reboot, the server may still have a TCB for the old=20
> connection
> and send you an ACK probe when you try to connect (if all=20
> goes well), and
> the client's TCP stack will RST and fail the connect. If things go
> not-so-well you have a packet filter somewhere inbetween that eats the
> ACK probe because its connection tracking engine thinks the connection
> is in half-open and shouldn't see any SYN-less ACKs yet.

i don't agree.  you're just making the bad behavior more rare by
choosing ports at random; you are not actually addressing the root
problem you describe.  is it possible to fix the RPC client to retry the
connection in this case?  and/or fix the server to recognize and remove
the context for the old connection?

introducing randomness here will make reproducing problems in this area
very difficult.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: xprt_bindresvport

[Olaf Kirch]

On Wed, Dec 08, 2004 at 06:33:15AM -0800, Lever, Charles wrote:
> > port by counting down from 800 to 0. I think this is a bug, because it
> > will potentially interfere with services trying to bind to 
> > low ports as
> > well.
> 
> is this idle speculation, or do you actually have a test case that
> fails?  :^)

I've seen many cases of glibc's bindresvport (which allocates from
600-1023) snatching ports used by other services, e.g. cups. The way
we solve this in user space is by having a file in /etc with a blacklist
of ports bindresvport isn't allowed to touch.

I admit that I haven't seen NFS step on someone else toes yet. But it
all depends on the number of mounts you have, and on coincidence.

At any rate, it really looks like a bug to me. Why should our scan include
ports in the low range (which is a no-no) but refuse to touch ports in the
800-1023 range? It's just an inverted test, IMO.

> i don't agree.  you're just making the bad behavior more rare by
> choosing ports at random; you are not actually addressing the root
> problem you describe.  is it possible to fix the RPC client to retry the
> connection in this case?  and/or fix the server to recognize and remove
> the context for the old connection?

You cannot fix the RPC client. It will not see any packets from the
server, so it simply sits there until it times out and retries the
connection. This can cause system boot to take quite a long time.

You can fix the firewall, this is currently being tossed around on the
netfilter list. Unfortunately not all routers out there run the latest and
greatest netfilter code; so we don't really have much control over that.

> introducing randomness here will make reproducing problems in this area
> very difficult.

I agree. And I would only enable randomization if we change the port
range to 600-up or something like this.

Olaf
-- 
Olaf Kirch     | Things that make Monday morning interesting, #2:
okir@suse.de   |        "We have 8,000 NFS mount points, why do we keep
---------------+ 	 running out of privileged ports?"


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs