RE: xprt_bindresvport

RE: xprt_bindresvport

[Lever, Charles]

> the current xprt_bindresvport implementation will search for=20
> a privileged
> port by counting down from 800 to 0. I think this is a bug, because it
> will potentially interfere with services trying to bind to=20
> low ports as
> well.

is this idle speculation, or do you actually have a test case that
fails?  :^)

> The bindresvport implementation in glibc picks from the 600-1023
> range.

we should review what other RPC implementations do (namely the reference
implementation, Solaris).

but also notice this cuts the usable port range in half (from ~800 to
~420).  we need some form of mitigation to ensure we aren't limiting the
number of NFS mounts a client can have.

> I also think it would be good to start at a "random" port. Otherwise,
> when you reboot, the server may still have a TCB for the old=20
> connection
> and send you an ACK probe when you try to connect (if all=20
> goes well), and
> the client's TCP stack will RST and fail the connect. If things go
> not-so-well you have a packet filter somewhere inbetween that eats the
> ACK probe because its connection tracking engine thinks the connection
> is in half-open and shouldn't see any SYN-less ACKs yet.

i don't agree.  you're just making the bad behavior more rare by
choosing ports at random; you are not actually addressing the root
problem you describe.  is it possible to fix the RPC client to retry the
connection in this case?  and/or fix the server to recognize and remove
the context for the old connection?

introducing randomness here will make reproducing problems in this area
very difficult.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[PATCH] xprt sharing (was Re: xprt_bindresvport)

[Mike Waychison]

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lever, Charles wrote:
>>the current xprt_bindresvport implementation will search for 
>>a privileged
>>port by counting down from 800 to 0. I think this is a bug, because it
>>will potentially interfere with services trying to bind to 
>>low ports as
>>well.
> 
> 
> is this idle speculation, or do you actually have a test case that
> fails?  :^)
> 

Well, I haven't seen this 'interfere' with services yet, I can imagine
that it is plausible for a service to want to grab some port at a later
time only to have it in use by nfs.

> 
>>The bindresvport implementation in glibc picks from the 600-1023
>>range.
> 
> 
> we should review what other RPC implementations do (namely the reference
> implementation, Solaris).
> 
> but also notice this cuts the usable port range in half (from ~800 to
> ~420).  we need some form of mitigation to ensure we aren't limiting the
> number of NFS mounts a client can have.


This has been bugging me for a while.  The fact that we are limitting
ourselves to a single nfs mount per port.  From what I can tell, Solaris
shares the transports between nfs mounts from the same server and saves
themselves a lot of trouble with running out of port numbers in doing so.

The attached patch does the same for Linux against 2.6.9.  We share
xprts from existing connections, effectively removing any limit on the
number of nfs mounts we have in the system.

The only thing to worry about now is any talking to the portmapper or
mountd from userspace using tcp, which will put the reserved ports in
TIME_WAIT state.  This can limit the 'speed' at which we mount many mounts.

- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE:  The opinions expressed in this email are held by me,
and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBt0VQdQs4kOxk3/MRAjo8AKCJjoqPxk1R3ev+o+UyquE0kiw0LQCfebi8
6mfhMSHLidFslVyt6jFeNis=
=0uCI
-----END PGP SIGNATURE-----

[-- Attachment #2: xprt_sharing.diff --]
[-- Type: text/x-patch, Size: 5311 bytes --]

This patch allows for sharing of xprts.  This is done by keeping a list of
current xprts and passing them back to the caller of xprt_create_proto if they
match the specifications required (IP X port X protocol X timeout).

We do this multiplexing at the xprt layer as it handles transport creation and
destruction.

This patch has been tested in a test-only environment but has been able to
handle a couple hundreds distinct nfs mounts from the same server over a single
tcp stream.

This effectively gets rid of the 800 nfs mounts max problem, as long as you
aren't mounting from many (800) nfs servers.

Signed-off-by: Mike Waychison <michael.waychison@sun.com>
Index: linux-2.6.9-nfs_portsharing/include/linux/sunrpc/xprt.h
===================================================================
--- linux-2.6.9-nfs_portsharing.orig/include/linux/sunrpc/xprt.h	2004-10-18 14:54:40.000000000 -0700
+++ linux-2.6.9-nfs_portsharing/include/linux/sunrpc/xprt.h	2004-10-26 13:22:41.000000000 -0700
@@ -15,6 +15,8 @@
 #include <linux/sunrpc/sched.h>
 #include <linux/sunrpc/xdr.h>
 
+#include <asm/atomic.h>
+
 /*
  * The transport code maintains an estimate on the maximum number of out-
  * standing RPC requests, using a smoothed version of the congestion
@@ -194,6 +196,9 @@ struct rpc_xprt {
 	void			(*old_write_space)(struct sock *);
 
 	wait_queue_head_t	cong_wait;
+
+	atomic_t		count;		/* shared xprt refcount */
+	struct list_head	shared;		/* link to shared list */
 };
 
 #ifdef __KERNEL__
Index: linux-2.6.9-nfs_portsharing/net/sunrpc/xprt.c
===================================================================
--- linux-2.6.9-nfs_portsharing.orig/net/sunrpc/xprt.c	2004-10-18 14:54:39.000000000 -0700
+++ linux-2.6.9-nfs_portsharing/net/sunrpc/xprt.c	2004-10-26 15:27:56.713490488 -0700
@@ -78,6 +78,12 @@
 #define XPRT_MAX_RESVPORT	(800)
 
 /*
+ * List of shared xprt
+ */
+static DECLARE_MUTEX(shared_xprt_sem);
+static LIST_HEAD(shared_xprt_list);
+
+/*
  * Local functions
  */
 static void	xprt_request_init(struct rpc_task *, struct rpc_xprt *);
@@ -1395,6 +1401,30 @@ xprt_release(struct rpc_task *task)
 }
 
 /*
+ * Compare two rpc_timeout to see if they are the same.
+ */
+static int
+xprt_is_same_timeout(struct rpc_timeout *left, struct rpc_timeout *right)
+{
+	return left->to_initval     == right->to_initval
+            && left->to_maxval      == right->to_maxval
+            && left->to_increment   == right->to_increment
+            && left->to_retries     == right->to_retries
+            && left->to_exponential == right->to_exponential;
+}
+/*
+ * Check to see if the timeout is the default timeout.
+ */
+static int
+xprt_is_default_timeout(struct rpc_timeout *to, int proto)
+{
+	struct rpc_timeout defaultto;
+
+	xprt_default_timeout(&defaultto, proto);
+	return xprt_is_same_timeout(&defaultto, to);
+}
+
+/*
  * Set default timeout parameters
  */
 void
@@ -1472,6 +1502,8 @@ xprt_setup(int proto, struct sockaddr_in
 	xprt->timer.data = (unsigned long) xprt;
 	xprt->last_used = jiffies;
 	xprt->port = XPRT_MAX_RESVPORT;
+	INIT_LIST_HEAD(&xprt->shared);
+	atomic_set(&xprt->count, 1);
 
 	/* Set timeout parameters */
 	if (to) {
@@ -1617,8 +1649,8 @@ failed:
 /*
  * Create an RPC client transport given the protocol and peer address.
  */
-struct rpc_xprt *
-xprt_create_proto(int proto, struct sockaddr_in *sap, struct rpc_timeout *to)
+static struct rpc_xprt *
+__xprt_create_proto(int proto, struct sockaddr_in *sap, struct rpc_timeout *to)
 {
 	struct rpc_xprt	*xprt;
 
@@ -1631,6 +1663,43 @@ xprt_create_proto(int proto, struct sock
 }
 
 /*
+ * Create an RPC client transport that is shared given the protocol and peer
+ * address.
+ */
+struct rpc_xprt *
+xprt_create_proto(int proto, struct sockaddr_in *sap, struct rpc_timeout *to)
+{
+	struct rpc_xprt *xprt;
+
+	down(&shared_xprt_sem);
+	/* walk the list and find an existing mathing xprt */
+	list_for_each_entry(xprt, &shared_xprt_list, shared) {
+		/* Filter out mismatches */
+		if (sap->sin_addr.s_addr != xprt->addr.sin_addr.s_addr)
+			continue;
+		if (sap->sin_port != xprt->addr.sin_port)
+			continue;
+		if (xprt->prot != proto)
+			continue;
+		if (to == NULL && !xprt_is_default_timeout(&xprt->timeout, proto))
+			continue;
+		if (to && !xprt_is_same_timeout(&xprt->timeout, to))
+			continue;
+
+		atomic_inc(&xprt->count);
+		goto out;
+	}
+
+	/* make a new one */
+	xprt = __xprt_create_proto(proto, sap, to);
+	if (!IS_ERR(xprt))
+		list_add(&xprt->shared, &shared_xprt_list);
+out:
+	up(&shared_xprt_sem);
+	return xprt;
+}
+
+/*
  * Prepare for transport shutdown.
  */
 void
@@ -1658,8 +1727,8 @@ xprt_clear_backlog(struct rpc_xprt *xprt
 /*
  * Destroy an RPC transport, killing off all requests.
  */
-int
-xprt_destroy(struct rpc_xprt *xprt)
+static int
+__xprt_destroy(struct rpc_xprt *xprt)
 {
 	dprintk("RPC:      destroying transport %p\n", xprt);
 	xprt_shutdown(xprt);
@@ -1670,3 +1739,20 @@ xprt_destroy(struct rpc_xprt *xprt)
 
 	return 0;
 }
+
+/*
+ * Destroy a shared RPC transport.
+ * (XXX: what about the remaining live requests?)
+ */
+int
+xprt_destroy(struct rpc_xprt *xprt)
+{
+	int ret = 0;
+	down(&shared_xprt_sem);
+	if (atomic_dec_and_test(&xprt->count)) {
+		list_del_init(&xprt->shared);
+		ret = __xprt_destroy(xprt);
+	}
+	up(&shared_xprt_sem);
+	return ret;
+}

Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)

[Olaf Kirch]

On Wed, Dec 08, 2004 at 01:17:53PM -0500, Mike Waychison wrote:
> This has been bugging me for a while.  The fact that we are limitting
> ourselves to a single nfs mount per port.  From what I can tell, Solaris
> shares the transports between nfs mounts from the same server and saves
> themselves a lot of trouble with running out of port numbers in doing so.

Shouldn't we allow NFS mounts to use non-privileged ports? Many
environments don't really care about the "security" provided by privileged
ports, but would be more than happy if they can run with a few hundred
NFS mounts

Olaf
-- 
Olaf Kirch     | Things that make Monday morning interesting, #2:
okir@suse.de   |        "We have 8,000 NFS mount points, why do we keep
---------------+ 	 running out of privileged ports?"


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)

[Trond Myklebust]

to den 09.12.2004 Klokka 12:31 (+0100) skreiv Olaf Kirch:

> Shouldn't we allow NFS mounts to use non-privileged ports? Many
> environments don't really care about the "security" provided by privileged
> ports, but would be more than happy if they can run with a few hundred
> NFS mounts

Most AUTH_SYS based models still require it, however I agree that use of
strong security makes the privileged port totally redundant.

Cheers,
  Trond
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)

[Olaf Kirch]

On Thu, Dec 09, 2004 at 08:36:55AM -0500, Trond Myklebust wrote:
> > Shouldn't we allow NFS mounts to use non-privileged ports? Many
> > environments don't really care about the "security" provided by privileged
> > ports, but would be more than happy if they can run with a few hundred
> > NFS mounts
> 
> Most AUTH_SYS based models still require it, however I agree that use of
> strong security makes the privileged port totally redundant.

The Linux auth_sys works fine with unprivileged ports if you allow it to;
so why shouldn't we make that configurable on the client too? It'd sure
help some  installations who for some reason or other habe an excessive
number of exported file systems (see .sig below :).

Olaf
-- 
Olaf Kirch     | Things that make Monday morning interesting, #2:
okir@suse.de   |        "We have 8,000 NFS mount points, why do we keep
---------------+ 	 running out of privileged ports?"


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)

[Trond Myklebust]

to den 09.12.2004 Klokka 14:44 (+0100) skreiv Olaf Kirch:

> The Linux auth_sys works fine with unprivileged ports if you allow it to;
> so why shouldn't we make that configurable on the client too? It'd sure
> help some  installations who for some reason or other habe an excessive
> number of exported file systems (see .sig below :).

_Another_ mount option? Urgh... 8-)

Seriously: if you have 8000 NFS mount points, you do not want 8000
different superblocks, 8000 different RPC client structs, and 8000
different sockets.
Apart from being a ridiculous waste of resources, that does little to
pop the contention bubble. It just ends up pushing it down to the
(shared) device layer.

So while I am open to the idea of making use of unprivileged ports, I do
not accept it as a substitute for socket sharing.

Cheers,
  Trond
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)

[Dan Stromberg]

[-- Attachment #1: Type: text/plain, Size: 1390 bytes --]

On Thu, 2004-12-09 at 12:31 +0100, Olaf Kirch wrote:
> On Wed, Dec 08, 2004 at 01:17:53PM -0500, Mike Waychison wrote:
> > This has been bugging me for a while.  The fact that we are limitting
> > ourselves to a single nfs mount per port.  From what I can tell, Solaris
> > shares the transports between nfs mounts from the same server and saves
> > themselves a lot of trouble with running out of port numbers in doing so.
> 
> Shouldn't we allow NFS mounts to use non-privileged ports? Many
> environments don't really care about the "security" provided by privileged
> ports, but would be more than happy if they can run with a few hundred
> NFS mounts

IMO, this is a good time to apply the principle: "Be liberal in what you
accept, and conservative in what you send".

Last I heard, windows didn't even have a concept of a reserved port.

When I wrote a BSD-compatible printsystem in python, I made it accept
connections from any port, but generate connections only from reserved
ports.

It'd probably be worthwhile to have options to make NFS (and my
printsystem) generate any port (not just reserved ones), and accept only
reserved ports - but the default probably should be to accept any port,
and send only reserved ports - not because reserved ports are effective
at all, but because it'll avoid never ending questions about why NFS
isn't working.



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)

[Trond Myklebust]

to den 09.12.2004 Klokka 11:34 (-0800) skreiv Dan Stromberg:

> It'd probably be worthwhile to have options to make NFS (and my
> printsystem) generate any port (not just reserved ones), and accept only
> reserved ports - but the default probably should be to accept any port,
> and send only reserved ports - not because reserved ports are effective
> at all, but because it'll avoid never ending questions about why NFS
> isn't working.

Sure. The questions will no longer read "why isn't NFS working". They'll
read "why can any Tom, Dick and Harry suddenly read my private email
directly from the NFS server?". 8-)

The standard AUTH_SYS/AUTH_UNIX authentication scheme only checks the
source IP address, and trusts the client 100% when it comes to supplying
the correct uid/gid/.... (see RFC1831). By placing the additional
requirement that the source must be a privileged port, one is at least
able to prevent ordinary users on an authorized client from being able
to spoof NFS requests.

Cheers,
  Trond

-- 
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] xprt sharing (was Re: xprt_bindresvport)

[Dan Stromberg]

[-- Attachment #1: Type: text/plain, Size: 1184 bytes --]

On Thu, 2004-12-09 at 16:33 -0500, Trond Myklebust wrote:
> to den 09.12.2004 Klokka 11:34 (-0800) skreiv Dan Stromberg:
> 
> > It'd probably be worthwhile to have options to make NFS (and my
> > printsystem) generate any port (not just reserved ones), and accept only
> > reserved ports - but the default probably should be to accept any port,
> > and send only reserved ports - not because reserved ports are effective
> > at all, but because it'll avoid never ending questions about why NFS
> > isn't working.
> 
> Sure. The questions will no longer read "why isn't NFS working". They'll
> read "why can any Tom, Dick and Harry suddenly read my private email
> directly from the NFS server?". 8-)
> 
> The standard AUTH_SYS/AUTH_UNIX authentication scheme only checks the
> source IP address, and trusts the client 100% when it comes to supplying
> the correct uid/gid/.... (see RFC1831). By placing the additional
> requirement that the source must be a privileged port, one is at least
> able to prevent ordinary users on an authorized client from being able
> to spoof NFS requests.
> 
> Cheers,
>   Trond

I bow before your superior geekiness.  :)



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: xprt_bindresvport

[Olaf Kirch]

On Wed, Dec 08, 2004 at 06:33:15AM -0800, Lever, Charles wrote:
> > port by counting down from 800 to 0. I think this is a bug, because it
> > will potentially interfere with services trying to bind to 
> > low ports as
> > well.
> 
> is this idle speculation, or do you actually have a test case that
> fails?  :^)

I've seen many cases of glibc's bindresvport (which allocates from
600-1023) snatching ports used by other services, e.g. cups. The way
we solve this in user space is by having a file in /etc with a blacklist
of ports bindresvport isn't allowed to touch.

I admit that I haven't seen NFS step on someone else toes yet. But it
all depends on the number of mounts you have, and on coincidence.

At any rate, it really looks like a bug to me. Why should our scan include
ports in the low range (which is a no-no) but refuse to touch ports in the
800-1023 range? It's just an inverted test, IMO.

> i don't agree.  you're just making the bad behavior more rare by
> choosing ports at random; you are not actually addressing the root
> problem you describe.  is it possible to fix the RPC client to retry the
> connection in this case?  and/or fix the server to recognize and remove
> the context for the old connection?

You cannot fix the RPC client. It will not see any packets from the
server, so it simply sits there until it times out and retries the
connection. This can cause system boot to take quite a long time.

You can fix the firewall, this is currently being tossed around on the
netfilter list. Unfortunately not all routers out there run the latest and
greatest netfilter code; so we don't really have much control over that.

> introducing randomness here will make reproducing problems in this area
> very difficult.

I agree. And I would only enable randomization if we change the port
range to 600-up or something like this.

Olaf
-- 
Olaf Kirch     | Things that make Monday morning interesting, #2:
okir@suse.de   |        "We have 8,000 NFS mount points, why do we keep
---------------+ 	 running out of privileged ports?"


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

xprt_bindresvport

[Olaf Kirch]

Hi,

the current xprt_bindresvport implementation will search for a privileged
port by counting down from 800 to 0. I think this is a bug, because it
will potentially interfere with services trying to bind to low ports as
well. The bindresvport implementation in glibc picks from the 600-1023
range.

I also think it would be good to start at a "random" port. Otherwise,
when you reboot, the server may still have a TCB for the old connection
and send you an ACK probe when you try to connect (if all goes well), and
the client's TCP stack will RST and fail the connect. If things go
not-so-well you have a packet filter somewhere inbetween that eats the
ACK probe because its connection tracking engine thinks the connection
is in half-open and shouldn't see any SYN-less ACKs yet.

Olaf
-- 
Olaf Kirch     | Things that make Monday morning interesting, #2:
okir@suse.de   |        "We have 8,000 NFS mount points, why do we keep
---------------+ 	 running out of privileged ports?"


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs