Questions and problems with NFS4

Questions and problems with NFS4

[J.A. Magallón]

Hi all...

I'm beginning to move my boxen to use nfs4, and have some questions...

First of all, versions of my software: kernel 2.6.33.5, nfs-utils 1.2.2
(perhaps some of my doubts are solved in newer versions)

Questions:

- Is it mandatory to use a setup with a common root for exported filesystems ?
  In my setup I first did not use it, no fsid=0 root, and everything
  worked fine, just change the mount type in the clients. In fact, new
  nfs-utils just try first version 4 and you dont have to do anything.
  You loose the ability to use simple paths inside /export, for example, 
  and have to use full paths like in V3, but that is not so bad.
  Am I loosing something else wrt functionality ?

- If you have a common root, is it mandatory to mount it in the client ?

- If I setup a common root (say /export), and aoutomount some folders:
  
  /scratch/local
  /scratch/node/00
  /scratch/node/01
  ......
  /export/scratch -> /scratch/local (bind mount).

  /scratch/node/?? is automounted from node ??, in which it is its own
  /scratch/local. When I acess scratch for local node, it backs to a local
  mount and mounts the REAL /scratch, not /export/scratch.
  Any ideas ?

- AFAIK, with NFS4 the only needed daemons are nfsd and idmapd. And the
  only accesible port from the outside is 2049, for nfsd.
  I have tried to strip down my nfs server (-N 2 -N 3 -U),
  but rpcinfo still gives me:

annwn:~# rpcinfo -p localhost
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  48461  status
    100024    1   tcp  37515  status
    100021    1   udp  38583  nlockmgr
    100021    3   udp  38583  nlockmgr
    100021    4   udp  38583  nlockmgr
    100021    1   tcp  37873  nlockmgr
    100021    3   tcp  37873  nlockmgr
    100021    4   tcp  37873  nlockmgr
    100003    4   tcp   2049  nfs
    100005    1   udp  45341  mountd
    100005    1   tcp  58639  mountd

  disabling portampper and mountd is just a matter of initscripts
  requirements, but how can I disable nlockmgr ? It isn't needed for
  NFS4, isn't it ? Nor portmapper nor mountd...

TIA

--
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free

Re: Questions and problems with NFS4

[Bian Naimeng]

> - If you have a common root, is it mandatory to mount it in the client ?
> 
> - If I setup a common root (say /export), and aoutomount some folders:
>   
>   /scratch/local
>   /scratch/node/00
>   /scratch/node/01
>   ......
>   /export/scratch -> /scratch/local (bind mount).
> 
>   /scratch/node/?? is automounted from node ??, in which it is its own
>   /scratch/local. When I acess scratch for local node, it backs to a local
>   mount and mounts the REAL /scratch, not /export/scratch.
>   Any ideas ?
> 

    I'm not sure i have understand your means, is it like this?

    # cat /etc/exports
      /nfsroot	*(fsid=0,insecure,no_root_squash,rw)

    # mkdir -p /nfsroot/nfsroot
    # touch /nfsroot/tstfile
    # service nfs restart
    # mount -t nfs 127.0.0.1:/nfsroot  /mnt

    # ls /mnt <== Here, we access real /nfsroot/nfsroot, not /nfsroot,
                  so we can not find "tstfile", right?

    Actually, when we use "-t nfs", nfs-utils try to mount NFSv4 first,
    if you want use NFSv3, you should use "-t nfs -o vers=3" in the nfs-utils-1.2.2.

> - AFAIK, with NFS4 the only needed daemons are nfsd and idmapd. And the
>   only accesible port from the outside is 2049, for nfsd.
>   I have tried to strip down my nfs server (-N 2 -N 3 -U),
>   but rpcinfo still gives me:
> 

    ... snip ...

>   disabling portampper and mountd is just a matter of initscripts
>   requirements, but how can I disable nlockmgr ? It isn't needed for
>   NFS4, isn't it ? Nor portmapper nor mountd...
> 

    As i see, there's no way to disabe nlockmgr, it's start by lockd,
    and lockd will be up when we start nfsd. It's always work until the
    last nfsd thread stop.

-- 
Best Regards
 Bian Naimeng

Re: Questions and problems with NFS4

[J. Bruce Fields]

On Wed, Jul 28, 2010 at 10:28:12AM +0800, Bian Naimeng wrote:
>     As i see, there's no way to disabe nlockmgr, it's start by lockd,
>     and lockd will be up when we start nfsd. It's always work until the
>     last nfsd thread stop.

Some day we should probably modify the server to turn off lockd in the
v4-only case.  For now we don't do that.

It should be fine to firewall off lockd if you only care about NFSv4.

--b.

Re: Questions and problems with NFS4

[Chuck Lever]

On 07/27/10 06:46 PM, J.A. Magallón wrote:
> - AFAIK, with NFS4 the only needed daemons are nfsd and idmapd. And the
>    only accesible port from the outside is 2049, for nfsd.
>    I have tried to strip down my nfs server (-N 2 -N 3 -U),
>    but rpcinfo still gives me:
>
> annwn:~# rpcinfo -p localhost
>     program vers proto   port  service
>      100000    4   tcp    111  portmapper
>      100000    3   tcp    111  portmapper
>      100000    2   tcp    111  portmapper
>      100000    4   udp    111  portmapper
>      100000    3   udp    111  portmapper
>      100000    2   udp    111  portmapper
>      100024    1   udp  48461  status
>      100024    1   tcp  37515  status
>      100021    1   udp  38583  nlockmgr
>      100021    3   udp  38583  nlockmgr
>      100021    4   udp  38583  nlockmgr
>      100021    1   tcp  37873  nlockmgr
>      100021    3   tcp  37873  nlockmgr
>      100021    4   tcp  37873  nlockmgr
>      100003    4   tcp   2049  nfs
>      100005    1   udp  45341  mountd
>      100005    1   tcp  58639  mountd
>
>    disabling portampper and mountd is just a matter of initscripts
>    requirements, but how can I disable nlockmgr ? It isn't needed for
>    NFS4, isn't it ? Nor portmapper nor mountd...

Strictly speaking, portmapper is not required for NFSv4 service. 
However, the NFS infrastructure on Linux is still designed for NFSv2 and 
v3.  There remains some work needed to make portmapper optional for a 
v4-only server.  For now, continue to run it in order to handle kernel 
upcalls.

rpc.mountd is, however, still required on Linux NFSv4 servers.  Although 
NFSv4 clients do not contact the server's mountd, the kernel's NFS 
server performs upcalls to rpc.mountd to manage export information.  You 
can firewall off the mountd service on the server without affecting 
NFSv4 clients.  Recent versions of rpc.mountd accept command line 
options that disable the mountd network service while still handling 
kernel upcalls.

And, as long as lockd is running, you will need to keep rpc.statd 
around.  Again, you can firewall this service so that it is not exposed 
on the network, but it must continue to be available to handle kernel 
upcalls.  This is something we hope to address eventually as part of the 
lockd work Bruce mentioned.

Re: Questions and problems with NFS4

[J.A. Magallón]

On Wed, 28 Jul 2010 14:08:08 -0400, Chuck Lever <chuck.lever@oracle.com> wrote:

> On 07/27/10 06:46 PM, J.A. Magallón wrote:
> > - AFAIK, with NFS4 the only needed daemons are nfsd and idmapd. And the
> >    only accesible port from the outside is 2049, for nfsd.
> >    I have tried to strip down my nfs server (-N 2 -N 3 -U),
> >    but rpcinfo still gives me:
> >
> > annwn:~# rpcinfo -p localhost
> >     program vers proto   port  service
> >      100000    4   tcp    111  portmapper
> >      100000    3   tcp    111  portmapper
> >      100000    2   tcp    111  portmapper
> >      100000    4   udp    111  portmapper
> >      100000    3   udp    111  portmapper
> >      100000    2   udp    111  portmapper
> >      100024    1   udp  48461  status
> >      100024    1   tcp  37515  status
> >      100021    1   udp  38583  nlockmgr
> >      100021    3   udp  38583  nlockmgr
> >      100021    4   udp  38583  nlockmgr
> >      100021    1   tcp  37873  nlockmgr
> >      100021    3   tcp  37873  nlockmgr
> >      100021    4   tcp  37873  nlockmgr
> >      100003    4   tcp   2049  nfs
> >      100005    1   udp  45341  mountd
> >      100005    1   tcp  58639  mountd
> >
> >    disabling portampper and mountd is just a matter of initscripts
> >    requirements, but how can I disable nlockmgr ? It isn't needed for
> >    NFS4, isn't it ? Nor portmapper nor mountd...
> 
> Strictly speaking, portmapper is not required for NFSv4 service. 
> However, the NFS infrastructure on Linux is still designed for NFSv2 and 
> v3.  There remains some work needed to make portmapper optional for a 
> v4-only server.  For now, continue to run it in order to handle kernel 
> upcalls.
> 
> rpc.mountd is, however, still required on Linux NFSv4 servers.  Although 
> NFSv4 clients do not contact the server's mountd, the kernel's NFS 
> server performs upcalls to rpc.mountd to manage export information.  You 
> can firewall off the mountd service on the server without affecting 
> NFSv4 clients.  Recent versions of rpc.mountd accept command line 
> options that disable the mountd network service while still handling 
> kernel upcalls.
> 
> And, as long as lockd is running, you will need to keep rpc.statd 
> around.  Again, you can firewall this service so that it is not exposed 
> on the network, but it must continue to be available to handle kernel 
> upcalls.  This is something we hope to address eventually as part of the 
> lockd work Bruce mentioned.

Err...

First of all, plz correct me if this is in some kind of wiki, web page or
the like, but I have not been able to find it. All this things should be
on a document somewhere, perhaps even in the kernel itself...
These are the things I have found out (thanks to your answers...).
I'm a long time unix admin, not new to NFS, but all this things were not
obvious to me, so perhaps they deserve to be in a document, something like
"NFS 3 to 4 migration for dummy admins":

- Everything just works if you mount shares as nfs4. Even, if you have a
  bunch of linux boxen with recent kernel/nfs-utils, probably you are
  already doing nfs4... modern mount tries nfs4 first.

- Using nfsroot with fsid=0 is not mandatory, nor bind-mounting everything
  under some /export (like many documents say), that only forces you to
  use the old way of specifying paths in the server (absolute, not
  relative to /export).

- Even if you use nfsroot(fsid=0), and you mount it on the client
  at /somepoint, you are not forced to mount everything else under
  /somepoint (plz, correct me if I'm wrong).

- You can strip your server for NFS4, but not too much... Old daemons
  are still needed locally. For example, you can get rid of NFS2 and
  UDP for nfsd (-N 2 -U, -any os still uses NFS2??-), and old
  mount protocols (-N 1 -N 2 for mountd)...

- ... but they can be firewalled, use is just local

- portmap/rpcbind is not needed, but still used because nfsd is not
  yet proper clean for only-nfs4-behavior.

There are also some things I have not been able to discover, like
using the interesting things of NFS4:

- How do you activate delegations ? Is this an automatic thing, or
  do I have to add any option somewhere ?

- How do you use cache ? Many docs talk about fsc option, but man
  does not mention it (nfs-utils 1.2.2).
  I have tried to use cachefiles module, and cachefilesd, but
  when I try to run it, I get:

bran:~/c/cachefilesd-0.10# ./cachefilesd -dddns -f $PWD/cachefilesd.conf
About to bind cache
CacheFiles bind failed: errno 95 (Operation not supported)

In dmesg:
[cachef] ==> cachefiles_get_security_ID({system_u:system_r:cachefiles_kernel_t:s0})
CacheFiles: Security denies permission to nominate security context: error -95

I can not run cachefiles without SELinux or something like that ?

Thanks, perhaps this notes help someone in the future. 

TIA

-- 
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free

Re: Questions and problems with NFS4

[J. Bruce Fields]

On Fri, Jul 30, 2010 at 02:18:37AM +0200, J.A. Magallón wrote:
> First of all, plz correct me if this is in some kind of wiki, web page or
> the like, but I have not been able to find it. All this things should be
> on a document somewhere, perhaps even in the kernel itself...

Agreed.

> These are the things I have found out (thanks to your answers...).
> I'm a long time unix admin, not new to NFS, but all this things were not
> obvious to me, so perhaps they deserve to be in a document, something like
> "NFS 3 to 4 migration for dummy admins":
> 
> - Everything just works if you mount shares as nfs4. Even, if you have a
>   bunch of linux boxen with recent kernel/nfs-utils, probably you are
>   already doing nfs4... modern mount tries nfs4 first.
> 
> - Using nfsroot with fsid=0 is not mandatory, nor bind-mounting everything
>   under some /export (like many documents say), that only forces you to

We should update that documentation.

>   use the old way of specifying paths in the server (absolute, not
>   relative to /export).
> 
> - Even if you use nfsroot(fsid=0), and you mount it on the client
>   at /somepoint, you are not forced to mount everything else under
>   /somepoint (plz, correct me if I'm wrong).
> 
> - You can strip your server for NFS4, but not too much... Old daemons
>   are still needed locally. For example, you can get rid of NFS2 and
>   UDP for nfsd (-N 2 -U, -any os still uses NFS2??-), and old
>   mount protocols (-N 1 -N 2 for mountd)...
> 
> - ... but they can be firewalled, use is just local
> 
> - portmap/rpcbind is not needed, but still used because nfsd is not
>   yet proper clean for only-nfs4-behavior.
> 
> There are also some things I have not been able to discover, like
> using the interesting things of NFS4:
> 
> - How do you activate delegations ? Is this an automatic thing, or
>   do I have to add any option somewhere ?

It's automatic.  You can turn them off on the server side with
/proc/sys/fs/leases-enable.  They'll be turned off automatically if the
server can't open a tcp connection back to the client (for example, if
the client is behind a NAT).

> - How do you use cache ? Many docs talk about fsc option, but man
>   does not mention it (nfs-utils 1.2.2).
>   I have tried to use cachefiles module, and cachefilesd, but
>   when I try to run it, I get:

I haven't used that.  I don't believe there's anything NFSv4-specific
about it.

--b.