Re: Massive NFS problems on large cluster with large number of mounts

Chuck Lever
2008-07-17T14:47:27Z

On Wed, Jul 16, 2008 at 3:06 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> On Wed, Jul 16, 2008 at 11:49:51AM +0200, Carsten Aulbert wrote:
>> Hi Trond et al.
>>
>> I'm following up on this discussion because we hit another problem:
>>
>> Trond Myklebust wrote:
>>
>> >
>> > Alternatively, just change the values of /proc/sys/sunrpc/min_resvport
>> > and /proc/sys/sunrpc/max_resvport to whatever range of ports you
>> > actually want to use.
>>
>> This works like a charm, however, if you set these values before
>> restarting the nfs-kernel-server then you are in deep trouble, since
>> when nfsd wants to start it needs to register with the portmapper, right?
>>
>> But what happens if this requests comes from a high^Wunpriviliged port?
>> Right:
>> Jul 16 11:46:43 d23 portmap[8216]: connect from 127.0.0.1 to set(nfs):
>> request from unprivileged port
>> Jul 16 11:46:43 d23 nfsd[8214]: nfssvc: writting fds to kernel failed:
>> errno 13 (Permission denied)
>> Jul 16 11:46:44 d23 kernel: [ 8437.726223] NFSD: Using
>> /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
>> Jul 16 11:46:44 d23 kernel: [ 8437.800607] NFSD: starting 90-second
>> grace period
>> Jul 16 11:46:44 d23 kernel: [ 8437.842891] nfsd: last server has exited
>> Jul 16 11:46:44 d23 kernel: [ 8437.879940] nfsd: unexporting all filesystems
>> Jul 16 11:46:44 d23 nfsd[8214]: nfssvc: Address already in use
>>
>>
>> Changing /proc/sys/sunrpc/max_resvport to 1023 again resolves this
>> issue, however defeats the purpose for the initial problem. I still need
>> to look into the code for hte portmapper, but is it easily possible that
>> the portmapper would accept nfsd requests from "insecure" ports also?
>> Since e are (mostly) in a controlled environment that should not pose a
>> problem.
>>
>> Anyone with an idea?
>
> The immediate problem seems like a kernel bug to me--it seems to me that
> the calls to local daemons should be ignoring {min_,max}_resvport. (Or
> is there some way the daemons can still know that those calls come from
> the local kernel?)

I tend to agree. The rpcbind client (at least) does specifically
require a privileged port, so a large min/max port range would be out
of the question for those rpc_clients.

--
Chuck Lever