Problems with Mac clients mounting a Linux server behind a firewall

Problems with Mac clients mounting a Linux server behind a firewall

[James Pearson]

I'm not absolutely sure if this is an NFS server issue, but I have an
Linux NFS server running CentOS4 that sits behind a firewall - about
once a week rpciod ends up using 99% CPU and the machine needs to be
rebooted

When it gets into this state, running 'netstat -u -a' shows thousands
of entries like:

udp        0      0 *:35071                     *:*
udp        0      0 *:34815                     *:*
udp        0      0 *:34559                     *:*
udp        0      0 *:34303                     *:*
udp        0      0 *:34047                     *:*
udp        0      0 *:33791                     *:*
udp        0      0 *:33535                     *:*
udp        0      0 *:33279                     *:*
udp        0      0 *:33151                     *:*
udp        0      0 *:32895                     *:*

and tcpdump shows lots of udp connection attempts to port 111 on
various Macs that are or have mounted the server (via an automounter).
The connections don't get through as they are blocked by the firewall.
None of these blocked connections are to Linux clients - it's just Mac
clients

Is there anything that NFS server-wise that could be cause the server
to attempt to contact clients in this way?

Thanks

James Pearson

Re: Problems with Mac clients mounting a Linux server behind a firewall

[James Pearson]

James Pearson wrote:
> I'm not absolutely sure if this is an NFS server issue, but I have an
> Linux NFS server running CentOS4 that sits behind a firewall - about
> once a week rpciod ends up using 99% CPU and the machine needs to be
> rebooted
> 
> When it gets into this state, running 'netstat -u -a' shows thousands
> of entries like:
> 
> udp        0      0 *:35071                     *:*
> udp        0      0 *:34815                     *:*
> udp        0      0 *:34559                     *:*
> udp        0      0 *:34303                     *:*
> udp        0      0 *:34047                     *:*
> udp        0      0 *:33791                     *:*
> udp        0      0 *:33535                     *:*
> udp        0      0 *:33279                     *:*
> udp        0      0 *:33151                     *:*
> udp        0      0 *:32895                     *:*
> 
> and tcpdump shows lots of udp connection attempts to port 111 on
> various Macs that are or have mounted the server (via an automounter).
> The connections don't get through as they are blocked by the firewall.
> None of these blocked connections are to Linux clients - it's just Mac
> clients
> 
> Is there anything that NFS server-wise that could be cause the server
> to attempt to contact clients in this way?

Running wireshark over the tcpdump output, all these portmap calls are like:

User Datagram Protocol, Src Port: 51947 (51947), Dst Port: sunrpc (111)
Remote Procedure Call, Type:Call XID:0xacd4150f
Portmap GETPORT Call NLM(100021) Version:4 UDP
     [Program Version: 2]
     [V2 Procedure: GETPORT (3)]
     Program: NLM (100021)
     Version: 4
     Proto: UDP (17)
     Port: 0

Why would an NFS server need to initiate a call to the lockd process on 
a (MacOS) client?

Thanks

James Pearson

Re: Problems with Mac clients mounting a Linux server behind a firewall

[J. Bruce Fields]

On Mon, Dec 22, 2008 at 04:53:57PM +0000, James Pearson wrote:
> James Pearson wrote:
>> I'm not absolutely sure if this is an NFS server issue, but I have an
>> Linux NFS server running CentOS4 that sits behind a firewall - about
>> once a week rpciod ends up using 99% CPU and the machine needs to be
>> rebooted
>>
>> When it gets into this state, running 'netstat -u -a' shows thousands
>> of entries like:
>>
>> udp        0      0 *:35071                     *:*
>> udp        0      0 *:34815                     *:*
>> udp        0      0 *:34559                     *:*
>> udp        0      0 *:34303                     *:*
>> udp        0      0 *:34047                     *:*
>> udp        0      0 *:33791                     *:*
>> udp        0      0 *:33535                     *:*
>> udp        0      0 *:33279                     *:*
>> udp        0      0 *:33151                     *:*
>> udp        0      0 *:32895                     *:*
>>
>> and tcpdump shows lots of udp connection attempts to port 111 on
>> various Macs that are or have mounted the server (via an automounter).
>> The connections don't get through as they are blocked by the firewall.
>> None of these blocked connections are to Linux clients - it's just Mac
>> clients
>>
>> Is there anything that NFS server-wise that could be cause the server
>> to attempt to contact clients in this way?
>
> Running wireshark over the tcpdump output, all these portmap calls are like:
>
> User Datagram Protocol, Src Port: 51947 (51947), Dst Port: sunrpc (111)
> Remote Procedure Call, Type:Call XID:0xacd4150f
> Portmap GETPORT Call NLM(100021) Version:4 UDP
>     [Program Version: 2]
>     [V2 Procedure: GETPORT (3)]
>     Program: NLM (100021)
>     Version: 4
>     Proto: UDP (17)
>     Port: 0
>
> Why would an NFS server need to initiate a call to the lockd process on  
> a (MacOS) client?

The communication between lockd's is 2-way, so that, for example,
servers can notify clients when locks are available.  Also, the server's
statd needs to contact the clients so it can ask the clients to notify
it when the clients reboot (and hence when their file locks should be
released).  And similarly the server needs to be able to notify the
clients when it reboots, so the client can reclaim any locks it
previously held.

So your firewall settings are a problem.  But, obviously, we should be
handling this more gracefully on the server.

--b.

Re: Problems with Mac clients mounting a Linux server behind a firewall

[James Pearson]

2008/12/22 J. Bruce Fields <bfields@fieldses.org>:
> On Mon, Dec 22, 2008 at 04:53:57PM +0000, James Pearson wrote:
>
>>> Is there anything that NFS server-wise that could be cause the server
>>> to attempt to contact clients in this way?
>>
>> Running wireshark over the tcpdump output, all these portmap calls are like:
>>
>> User Datagram Protocol, Src Port: 51947 (51947), Dst Port: sunrpc (111)
>> Remote Procedure Call, Type:Call XID:0xacd4150f
>> Portmap GETPORT Call NLM(100021) Version:4 UDP
>>     [Program Version: 2]
>>     [V2 Procedure: GETPORT (3)]
>>     Program: NLM (100021)
>>     Version: 4
>>     Proto: UDP (17)
>>     Port: 0
>>
>> Why would an NFS server need to initiate a call to the lockd process on
>> a (MacOS) client?
>
> The communication between lockd's is 2-way, so that, for example,
> servers can notify clients when locks are available.  Also, the server's
> statd needs to contact the clients so it can ask the clients to notify
> it when the clients reboot (and hence when their file locks should be
> released).  And similarly the server needs to be able to notify the
> clients when it reboots, so the client can reclaim any locks it
> previously held.
>
> So your firewall settings are a problem.  But, obviously, we should be
> handling this more gracefully on the server.

Thanks for the explanation - however, I never come across this
requirement before with NFS servers behind firewalls - for example,
RedHat's documentation at <http://kbase.redhat.com/faq/docs/DOC-3259>
says the usual stuff about fixing the ports of various portmap
services for the server and opening those ports for traffic to the
server through the firewall - but no mention of the server needing to
talk to the portmapper and lockd on the clients ... and I can't see
how this would work as the portmapper port and any possible lockd port
will have to be open on the firewall from server to all clients ...

Also, it is strange that this problem only happens with MacOS clients
- I've never see the server trying to talk to Linux clients in this
way. So I guess it is something that the MacOS clients do differently
...

Is there any way I can configure the server to ignore locking
completely? e.g. turn off statd?

Thanks

James Pearson

Re: Problems with Mac clients mounting a Linux server behind a firewall

[J. Bruce Fields]

On Mon, Dec 22, 2008 at 08:36:12PM +0000, James Pearson wrote:
> 2008/12/22 J. Bruce Fields <bfields@fieldses.org>:
> > On Mon, Dec 22, 2008 at 04:53:57PM +0000, James Pearson wrote:
> >
> >>> Is there anything that NFS server-wise that could be cause the server
> >>> to attempt to contact clients in this way?
> >>
> >> Running wireshark over the tcpdump output, all these portmap calls are like:
> >>
> >> User Datagram Protocol, Src Port: 51947 (51947), Dst Port: sunrpc (111)
> >> Remote Procedure Call, Type:Call XID:0xacd4150f
> >> Portmap GETPORT Call NLM(100021) Version:4 UDP
> >>     [Program Version: 2]
> >>     [V2 Procedure: GETPORT (3)]
> >>     Program: NLM (100021)
> >>     Version: 4
> >>     Proto: UDP (17)
> >>     Port: 0
> >>
> >> Why would an NFS server need to initiate a call to the lockd process on
> >> a (MacOS) client?
> >
> > The communication between lockd's is 2-way, so that, for example,
> > servers can notify clients when locks are available.  Also, the server's
> > statd needs to contact the clients so it can ask the clients to notify
> > it when the clients reboot (and hence when their file locks should be
> > released).  And similarly the server needs to be able to notify the
> > clients when it reboots, so the client can reclaim any locks it
> > previously held.
> >
> > So your firewall settings are a problem.  But, obviously, we should be
> > handling this more gracefully on the server.
> 
> Thanks for the explanation - however, I never come across this
> requirement before with NFS servers behind firewalls - for example,
> RedHat's documentation at <http://kbase.redhat.com/faq/docs/DOC-3259>
> says the usual stuff about fixing the ports of various portmap
> services for the server and opening those ports for traffic to the
> server through the firewall - but no mention of the server needing to
> talk to the portmapper and lockd on the clients ... and I can't see
> how this would work as the portmapper port and any possible lockd port
> will have to be open on the firewall from server to all clients ...
> 
> Also, it is strange that this problem only happens with MacOS clients
> - I've never see the server trying to talk to Linux clients in this
> way. So I guess it is something that the MacOS clients do differently

There may be a bug keeping the server from clearing out old statd
entries?  If so perhaps it's just that every client only gets monitored
once, and that that happened longer ago for the linux clients than the
mac clients?  Our recovery code needs some work.

I can't think of any reason the client implementation would matter,
unless perhaps the mac clients just aren't doing monitored locking at
all.

> Is there any way I can configure the server to ignore locking
> completely? e.g. turn off statd?

That might work.  It might end up giving the applications that are using
locking unexpected errors.

Better might be to have the clients to mount with the "nolock" option.
Then the client will pretend to do locking but without actually
acquiring locks on the server, which may of course be dangerous
depending on how the locks are being used....

--b.