[NFS] nfs locking problems maybe solved

[NFS] nfs locking problems maybe solved

[Erik Hensema / HostingXS Internet Services]

Hi all,

I may have solved the locking problems I experience every once in a 
while on my machines. It seems the nfs server needs access to the 
portmapper on the clients. So, I granted access in hosts.allow and 
that seems to have solved a lot of problems.

I experienced two kinds of locking problems:

1) sometimes rrdtool could not lock rrd files

rrdtool runs every 5 minutes on all clients, this problem always 
solved itself

2) sometimes it was impossible to lock any file on an entire 
filesystem and it would be impossible to release those locks even by 
unmounting the filesystem and mounting it again


This morning I started receiving mails from all client machines 
complaining they couldn't lock rrd files. The nfs server was spewing 
the following lines in its log:

May 21 09:25:02 ananke kernel: lockd: server voyager not responding, 
timed out
May 21 09:25:02 ananke kernel: lockd: couldn't create RPC handle for 
voyager

Meanwhile the client said:
May 21 09:09:25 voyager portmap[13799]: connect from 192.168.200.45 to 
getport(nlockmgr): request from unauthorized host

So I granted 192.168.200.45 access to the portmapper on voyager, and 
most clients were unstuck. I updated hosts.allow on most clients to 
grant access to the portmapper now.

Now I still have about 4 clients which can't lock rrd files. Umounting 
the filesystem doesn't help. Umounting the filesystem for about 10 
minutes doesn't help. So, I remounted the fs nolock on these 
machines.

So, now I've got problem #2 on these clients on the filesystem 
containing rrd files. Never had that before.

Problem #2 was always on the mail spool. Not sure if the problem is 
still there, however currently the clients do seem to be able to lock 
mailboxes.

I think problem #1 is gone now, I haven't received any mails of 
servers not being able to lock rrd files anymore, except for the 4 
clients which are never ever able to lock those files.


So, in short: locking is still unreliable for me, but the situation 
has improved.

Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 
2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).


Met vriendelijke groet,


Erik Hensema
-- 
HostingXS
eXcellent Service

Telefoon: 024 - 324 91 77
E-mail:   hensema-yolSOB3tinz/D1n+0JDH9g@public.gmane.org
Website:  http://www.hostingxs.nl/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs locking problems maybe solved

[Talpey, Thomas]

At 07:20 AM 5/21/2008, Erik Hensema / HostingXS Internet Services wrote:
>So I granted 192.168.200.45 access to the portmapper on voyager, and 
>most clients were unstuck. I updated hosts.allow on most clients to 
>grant access to the portmapper now.
>
>Now I still have about 4 clients which can't lock rrd files.
...
>
>Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 
>2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).

You need to grant access to the client's portmap, nlm and statd ports
in order for the server to call back with locking operations. Unfortunately,
only portmap is at a well-known port (111). The nlm and statd ports are
dynamically selected, and receive random high numbered port values.

They are then advertised in portmap so if you are concerned about opening
up an entire port range to the servers, you can see these with "rpcinfo -p".
This command will show them under the names "nlockmgr" and "status"
respectively. Remember these ports may change at every boot.

Also be aware that "statd" is a kernel process on opensuse. Most other
distros have rpc.statd as a user process.

Tom.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs locking problems maybe solved

[Talpey, Thomas]

Sorry - to be clear, the "rpcinfo -p" command must be run on the *client*,
or directed to the client with "rpcinfo -p <clienthostname>", in order to
show the client's current ports.

Tom.

At 07:58 AM 5/21/2008, Talpey, Thomas wrote:
>At 07:20 AM 5/21/2008, Erik Hensema / HostingXS Internet Services wrote:
>>So I granted 192.168.200.45 access to the portmapper on voyager, and 
>>most clients were unstuck. I updated hosts.allow on most clients to 
>>grant access to the portmapper now.
>>
>>Now I still have about 4 clients which can't lock rrd files.
>...
>>
>>Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 
>>2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
>
>You need to grant access to the client's portmap, nlm and statd ports
>in order for the server to call back with locking operations. Unfortunately,
>only portmap is at a well-known port (111). The nlm and statd ports are
>dynamically selected, and receive random high numbered port values.
>
>They are then advertised in portmap so if you are concerned about opening
>up an entire port range to the servers, you can see these with "rpcinfo -p".
>This command will show them under the names "nlockmgr" and "status"
>respectively. Remember these ports may change at every boot.
>
>Also be aware that "statd" is a kernel process on opensuse. Most other
>distros have rpc.statd as a user process.
>
>Tom.
>
>
>-------------------------------------------------------------------------
>This SF.net email is sponsored by: Microsoft 
>Defy all challenges. Microsoft(R) Visual Studio 2008. 
>http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>_______________________________________________
>NFS maillist  -  NFS@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/nfs
>_______________________________________________
>Please note that nfs@lists.sourceforge.net is being discontinued.
>Please subscribe to linux-nfs@vger.kernel.org instead.
>    http://vger.kernel.org/vger-lists.html#linux-nfs


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs locking problems maybe solved

[Jeff Layton]

On Wed, 21 May 2008 07:58:13 -0400
"Talpey, Thomas" <Thomas.Talpey@netapp.com> wrote:

> At 07:20 AM 5/21/2008, Erik Hensema / HostingXS Internet Services wrote:
> >So I granted 192.168.200.45 access to the portmapper on voyager, and 
> >most clients were unstuck. I updated hosts.allow on most clients to 
> >grant access to the portmapper now.
> >
> >Now I still have about 4 clients which can't lock rrd files.
> ...
> >
> >Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 
> >2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
> 
> You need to grant access to the client's portmap, nlm and statd ports
> in order for the server to call back with locking operations. Unfortunately,
> only portmap is at a well-known port (111). The nlm and statd ports are
> dynamically selected, and receive random high numbered port values.
> 

> They are then advertised in portmap so if you are concerned about opening
> up an entire port range to the servers, you can see these with "rpcinfo -p".
> This command will show them under the names "nlockmgr" and "status"
> respectively. Remember these ports may change at every boot.
> 
> Also be aware that "statd" is a kernel process on opensuse. Most other
> distros have rpc.statd as a user process.
> 

It's also possible to statically set ports for each of these services
(though I'm not certain about SUSE's in-kernel statd). This is what you
want to be doing if you're going to have firewalls between NFS servers
and clients. With RH/Fedora distros, this is generally set up
in /etc/sysconfig/nfs, but I'm not sure how SUSE does it...

-- 
Jeff Layton <jlayton@redhat.com>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs locking problems maybe solved

[Erik Hensema / HostingXS Internet Services]

On woensdag 21 mei 2008, Talpey, Thomas wrote:
> At 07:20 AM 5/21/2008, Erik Hensema / HostingXS Internet Services 
wrote:
> >So I granted 192.168.200.45 access to the portmapper on voyager,
> > and most clients were unstuck. I updated hosts.allow on most
> > clients to grant access to the portmapper now.
> >
> >Now I still have about 4 clients which can't lock rrd files.
>
> ...
>
> >Everything is NFSv3 by the way. Most clients are opensuse 10.2
> > (kernel 2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
>
> You need to grant access to the client's portmap, nlm and statd
> ports in order for the server to call back with locking operations.
> Unfortunately, only portmap is at a well-known port (111). The nlm
> and statd ports are dynamically selected, and receive random high
> numbered port values.

That's no problem to me, I don't firewall ports on the internal 
network. I do have an "ALL : ALL" line in hosts.deny, so I 
explicitely have to open services using tcp_wrapper.


Met vriendelijke groet,


Erik Hensema
-- 
HostingXS
eXcellent Service

Telefoon: 024 - 324 91 77
E-mail:   hensema-yolSOB3tinz/D1n+0JDH9g@public.gmane.org
Website:  http://www.hostingxs.nl/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs locking problems maybe solved

[Talpey, Thomas]

At 09:27 AM 5/21/2008, Erik Hensema / HostingXS Internet Services wrote:
>That's no problem to me, I don't firewall ports on the internal 
>network. I do have an "ALL : ALL" line in hosts.deny, so I 
>explicitely have to open services using tcp_wrapper.

Your followup reminds me that some kernels use local RPC to communicate
between nlm and statd on the client as well. So, if you haven't included
the local client host in your hosts.allow for at least these services, that
may be advisable.

Tom.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs locking problems maybe solved

[Suresh Jayaraman]

Talpey, Thomas wrote:
> At 07:20 AM 5/21/2008, Erik Hensema / HostingXS Internet Services wrote:
>> So I granted 192.168.200.45 access to the portmapper on voyager, and 
>> most clients were unstuck. I updated hosts.allow on most clients to 
>> grant access to the portmapper now.
>>
>> Now I still have about 4 clients which can't lock rrd files.
> ...
>> Everything is NFSv3 by the way. Most clients are opensuse 10.2 (kernel 
>> 2.6.18.8), server is opensuse 10.3 (kernel 2.6.22.17).
> 
> You need to grant access to the client's portmap, nlm and statd ports
> in order for the server to call back with locking operations. Unfortunately,
> only portmap is at a well-known port (111). The nlm and statd ports are
> dynamically selected, and receive random high numbered port values.
> 
> They are then advertised in portmap so if you are concerned about opening
> up an entire port range to the servers, you can see these with "rpcinfo -p".
> This command will show them under the names "nlockmgr" and "status"
> respectively. Remember these ports may change at every boot.
> 
> Also be aware that "statd" is a kernel process on opensuse. Most other
> distros have rpc.statd as a user process.
> 

"statd" is in-kernel only till OpenSUSE 10.2 and not in OpenSUSE 10.3.
It's a userspace process in 10.3 and make sure it's running.


Thanks,

-- 

Suresh Jayaraman

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs