Re: [PATCH 1/1] NFSv4.0 allow nconnect for v4.0

[Trond Myklebust <trondmy@hammerspace.com>]

On Mon, 2020-01-27 at 13:39 -0500, Steve Dickson wrote:
> 
> On 1/22/20 7:23 PM, Trond Myklebust wrote:
> > On Mon, 2020-01-20 at 10:35 -0500, Steve Dickson wrote:
> > > Hello,
> > > 
> > > On 1/16/20 2:08 PM, Olga Kornievskaia wrote:
> > > > From: Olga Kornievskaia <kolga@netapp.com>
> > > > 
> > > > Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> > > > ---
> > > >  fs/nfs/nfs4client.c | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
> > > > index 460d625..4df3fb0 100644
> > > > --- a/fs/nfs/nfs4client.c
> > > > +++ b/fs/nfs/nfs4client.c
> > > > @@ -881,7 +881,7 @@ static int nfs4_set_client(struct
> > > > nfs_server
> > > > *server,
> > > >  
> > > >  	if (minorversion == 0)
> > > >  		__set_bit(NFS_CS_REUSEPORT,
> > > > &cl_init.init_flags);
> > > > -	else if (proto == XPRT_TRANSPORT_TCP)
> > > > +	if (proto == XPRT_TRANSPORT_TCP)
> > > >  		cl_init.nconnect = nconnect;
> > > >  
> > > >  	if (server->flags & NFS_MOUNT_NORESVPORT)
> > > > 
> > > Tested-by: Steve Dickson <steved@redhat.com>
> > > 
> > > With this patch v4.0 mounts act just like v4.1/v4.2 mounts
> > > But is that a good thing. :-)  
> > > 
> > > Here is what I've found in my testing...
> > > 
> > > mount -onconnect=12 172.31.1.54:/home/tmp /mnt/tmp
> > > 
> > > Will create 12 TCP connections and maintain those 12 
> > > connections until the umount happens. By maintain I mean 
> > > if the connection times out, it is reconnected 
> > > to maintain the 12 connections 
> > > 
> > > # mount -onconnect=12 172.31.1.54:/home/tmp /mnt/tmp
> > > # netstat -an | grep 172.31.1.54 | wc -l
> > > 12
> > > # netstat -an | grep 172.31.1.54        
> > > tcp        0      0
> > > 172.31.1.24:901         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:667         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:746         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:672         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:832         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:895         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:673         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:732         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:795         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:918         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:674         172.31.1.54:2049        ESTABLISHED
> > > tcp        0      0
> > > 172.31.1.24:953         172.31.1.54:2049        ESTABLISHED
> > > 
> > > # umount /mnt/tmp
> > > # netstat -an | grep 172.31.1.54 | wc -l
> > > 12
> > > # netstat -an | grep 172.31.1.54
> > > tcp        0      0
> > > 172.31.1.24:901         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:667         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:746         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:672         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:832         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:895         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:673         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:732         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:795         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:918         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:674         172.31.1.54:2049        TIME_WAIT  
> > > tcp        0      0
> > > 172.31.1.24:953         172.31.1.54:2049        TIME_WAIT 
> > > 
> > > Is this the expected behavior? 
> > > 
> > > If so I have a few concerns...
> > > 
> > > * The connections walk all over the /etc/services namespace.
> > > Meaning
> > > using ports that are reserved for registered services, something
> > > we've tried to avoid in userland by not binding to privilege
> > > ports
> > > and
> > > use of backlist ports via /etc/bindresvport.blacklist
> > > 
> > > * When the unmount happens, all those connections go into
> > > TIME_WAIT
> > > on 
> > > privilege ports and there are only so many of those. Not good
> > > during
> > > mount 
> > > storms (when a server reboots and thousand of home dirs are
> > > remounted).
> > > 
> > > * No man page describing the new feature.
> > > 
> > > I realize there is not much we can do about some of these
> > > (aka umount==>TIME_WAIT) but I think we need to document 
> > > what we are doing to people's connection namespace when 
> > > they use this feature. 
> > 
> > I'm not sure that I understand the concern. The connections are
> > limited
> > to a specific window of ports by the min_resvport/max_resvport
> > sunrpc
> > module parameters just as they were before we added 'nconnect'.
> > Nothing
> > has changed in the way we choose ports...
> > 
> Maybe this problem has existed for a while... 
> 
> Here are the mins/max ports
> RPC_DEF_MIN_RESVPORT    (665U)
> RPC_DEF_MAX_RESVPORT    (1023U)
> 
> From /etc/services there are the services in that range
> acp(674), ha-cluster(694), kerberos-adm(749), kerberos-iv(750)
> webster(765), phonebook(767), rsync(873), rquotad(875), 
> telnets(992), imaps(993), pop3s(995)
> 
> Granted a lot of these are unused/legacy services, but some of
> them, like imaps and rsync, are still used. 
> 
> My point is since the nconnect connections are persistent, for
> the life of the mount, we could end up squatting on ports other
> services will needed.
> 
> Maybe there is not much we can do about this... But we should explain
> somewhere, like the man page, that nconnect will create up to 16
> persistent connection on register privilege ports.


If users have a need to run servers on a port that might chosen by a
kernel nfs, lockd or rpcbind client, then they can guarantee no
collisions by redefining the 'privileged ports' available to sunrpc to
any arbitrary range <portnr1> - <portnr2>:

Either

 * Reserve the ports at module load time, by adding a line to a config
   file /etc/modprobe.d/foo.conf of the form
      options sunrpc min_resvport=<portnr1> max_resvport=<portnr2>

or

 * Change the port reservation after module load (but before mounting
   your first NFS filesystem) using
      echo '<portnr1>' > /sys/module/sunrpc/parameters/min_resvport
      echo '<portnr2>' > /sys/module/sunrpc/parameters/max_resvport

This is something users ought to be doing already if they need
guaranteed availability of specific ports in the range 665-1023 while
using the NFS client. That is entirely independent of whether or not
they are using nconnect.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com