Re: Massive NFS problems on large cluster with large number of mounts

["J. Bruce Fields" <bfields@fieldses.org>]

On Wed, Jul 16, 2008 at 11:49:51AM +0200, Carsten Aulbert wrote:
> Hi Trond et al.
> 
> I'm following up on this discussion because we hit another problem:
> 
> Trond Myklebust wrote:
> 
> > 
> > Alternatively, just change the values of /proc/sys/sunrpc/min_resvport
> > and /proc/sys/sunrpc/max_resvport to whatever range of ports you
> > actually want to use.
> 
> This works like a charm, however, if you set these values before
> restarting the nfs-kernel-server then you are in deep trouble, since
> when nfsd wants to start it needs to register with the portmapper, right?
> 
> But what happens if this requests comes from a high^Wunpriviliged port?
> Right:
> Jul 16 11:46:43 d23 portmap[8216]: connect from 127.0.0.1 to set(nfs):
> request from unprivileged port
> Jul 16 11:46:43 d23 nfsd[8214]: nfssvc: writting fds to kernel failed:
> errno 13 (Permission denied)
> Jul 16 11:46:44 d23 kernel: [ 8437.726223] NFSD: Using
> /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
> Jul 16 11:46:44 d23 kernel: [ 8437.800607] NFSD: starting 90-second
> grace period
> Jul 16 11:46:44 d23 kernel: [ 8437.842891] nfsd: last server has exited
> Jul 16 11:46:44 d23 kernel: [ 8437.879940] nfsd: unexporting all filesystems
> Jul 16 11:46:44 d23 nfsd[8214]: nfssvc: Address already in use
> 
> 
> Changing /proc/sys/sunrpc/max_resvport to 1023 again resolves this
> issue, however defeats the purpose for the initial problem. I still need
> to look into the code for hte portmapper, but is it easily possible that
> the portmapper would accept nfsd requests from "insecure" ports also?
> Since e are (mostly) in a controlled environment that should not pose a
> problem.
> 
> Anyone with an idea?

The immediate problem seems like a kernel bug to me--it seems to me that
the calls to local daemons should be ignoring {min_,max}_resvport.  (Or
is there some way the daemons can still know that those calls come from
the local kernel?)

--b.