Re: [PATCH/RFC: nfs-utils] Common systemd unit files for nfs-utils.

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 17889 bytes --]

On Thu, 30 Jan 2014 15:06:34 -0500 Steve Dickson <SteveD@redhat.com> wrote:

> 
> 
> On 01/30/2014 01:24 AM, NeilBrown wrote:
> > 
> > Hi all,
> > 
> >  I would really like to see a common set of systemd unit files used for
> >  nfs-utils by every distribution (that actually uses systemd), and would like
> >  those unit files to be in the upstream nfs-utils package.
> > 
> >  To that end I have put together the following collection of unit files which
> >  seem right to me, and appear to work as I think I want them to work.
> > 
> >  I've looked at the unit files in Fedora and borrowed some ideas while
> >  discarding and changing others.  I won't try to list and justify all the
> >  changes here, but I'm perfectly willing (possibly even eager) to justify
> >  anything specific that anyone cares to ask about.
> > 
> >  So:
> >   1/ Do you agree that a collection of systemd unit files belongs in
> >      nfs-utils?
> I would think so... 
> >   2/ Do you think it reasonable to expect most (systemd using) distros to
> >      use the one set?  I will certainly try to ensure openSUSE does if
> >      upstream accepts them.
> >   3/ Do you have any comments/question about those below?
> > 
> > 
> > Thanks,
> > NeilBrown
> > 
> > diff --git a/systemd/README b/systemd/README
> > new file mode 100644
> > index 000000000000..f0fb68825499
> > --- /dev/null
> > +++ b/systemd/README
> > @@ -0,0 +1,50 @@
> > +
> > +Notes about systemd unit files for nfs-utils.
> > +
> > +The unit files provided here should be sufficient for systemd
> > +to manage all daemons and related services provides by nfs-utils.
> > +
> > +They do *not* include any unit files for separate services such as
> > +rpc.rquotad (in the 'quota' package) or rpcbind.
> > +
> > +There are 4 units that can be 'enabled' or 'disabled' by systemctl, or
> > +by a suitable 'preset' setting:
> > +
> > + nfs-server.target
> > +    If enabled, nfs service is started together with dependencies
> > +    such as mountd, statd, rpc.idmapd
> > +
> > + nfs-client.target
> > +    If enabled, daemons needs for an nfs client are enabled.
> > +    This does *not* include rpc.statd.  the rpc-statd.service unit
> > +    is started by /usr/sbin/start-statd which mount.nfs will run
> > +    if statd is needed.
> > +
> > + nfs-secure.target
> > +    If enabled, then rpc.gssd will be run when either -client or
> > +    -server is started, and rpc.svcgssd will be run when -server
> > +    is started
> > +
> > + nfs-blkmap.target
> > +    If enabled, then blkmapd will be run when nfs-client.target is
> > +    started.
> Do we really need all of these targets? Could we cut it down to
> two. nfs.target and nfs-secure.target?
> 

Excellent question.
My thought was to err on the side of providing too many rather than too few.
If a distro wants to remove a particular choice it can use the "presets"
feature off systemd to make sure that target is always enabled.

But I think you might be questioning specific targets rather than the general
number of targets - I respond to that below.


> > +
> > +
> > +It is possible that we should have an nfs-statd.target which can
> > +selectively enable statd being stared by -server and sm-notify
> > +being started by -server or -client.  That way it could be disabled
> > +completely on V4-only configurations.  Currently statd is always
> > +started on the server and sm-notify is always run if server or
> > +client is enabled.
> > +
> > +Stopping nfs-server will also stop rpc.mountd, and rpc.svcgssd.
> > +It cannot stop rpc.statd or rpc.gssd as they may be in use by the
> > +client and systemd cannot specify is two-pronged reverse dependency.
> > +(i.e. stop this unit if none of these units are running)
> > +
> > +Distro specific commandline configuration can be provided by
> > +installing a script /usr/lib/systemd/scripts/nfs-utils_env.sh
> > +This should write /run/sysconfig/nfs-utils based on configuration
> > +information such as in /etc/sysconfig/nfs or /etc/defaults/nfs.
> > +It should write to a tmp file and rename to the target to
> > +avoid parallel units seeing incomplete copies of the file.
> 
> > diff --git a/systemd/nfs-blkmap.service b/systemd/nfs-blkmap.service
> > new file mode 100644
> > index 000000000000..7319a88661cc
> > --- /dev/null
> > +++ b/systemd/nfs-blkmap.service
> > @@ -0,0 +1,11 @@
> > +[Unit]
> > +Description=pNFS block layout mapping daemon
> > +After=var-lib-nfs-rpc_pipefs.mount
> > +Requires=var-lib-nfs-rpc_pipefs.mount
> > +
> > +Requisite=nfs-blkmap.target
> > +After=nfs-blkmap.target
> > +
> > +[Service]
> > +Type=forking
> > +ExecStart=/usr/sbin/blkmapd
> Is this even supported anymore? Maybe we can just drop it???

No idea.  I found it as I was looking around and assumed it should be started.
If it is no longer supported, how does NFS now find the block devices for
direct pnfs access?


> 
> 
> > diff --git a/systemd/nfs-blkmap.target b/systemd/nfs-blkmap.target
> > new file mode 100644
> > index 000000000000..fbcc111152ee
> > --- /dev/null
> > +++ b/systemd/nfs-blkmap.target
> > @@ -0,0 +1,8 @@
> > +[Unit]
> > +Description= PNFS blkmaping enablement.
> > +# If this target is enabled, then blkmapd will be started
> > +# as required.  If it is not enabled it won't.
> > +
> > +[Install]
> > +WantedBy=remote-fs.target
> > +WantedBy=multi-user.target
> > \ No newline at end of file
> 
> Again, why is a client target needed? Now that idmappings are
> stored in the key ring what needs to be started on the client?

rpc.gssd?

Also sm-notify needs to be run on a machine this is used as an NFS client.
This should happen even if no NFS filesystems are mounted at boot.
An 'nfs-client' target seems the natural place to put that requirement.

> 
> > diff --git a/systemd/nfs-client.target b/systemd/nfs-client.target
> > new file mode 100644
> > index 000000000000..fa591354abf3
> > --- /dev/null
> > +++ b/systemd/nfs-client.target
> > @@ -0,0 +1,13 @@
> > +[Unit]
> > +Description=NFS client services
> > +Before=remote-fs-pre.target
> > +Wants=remote-fs-pre.target
> > +
> > +# Note: we don't "Wants=rpc-statd.service" as "mount.nfs" will arrange to
> > +# start that on demand if needed.
> > +Wants=rpc-gssd.service nfs-blkmap.service rpc-statd-notify.service
> > +Before=rpc-gssd.service nfs-blkmap.service
> > +
> > +[Install]
> > +WantedBy=multi-user.target
> > +WantedBy=remote-fs.target
> 
> 
> > diff --git a/systemd/nfs-idmapd.service b/systemd/nfs-idmapd.service
> > new file mode 100644
> > index 000000000000..6c2e1537f064
> > --- /dev/null
> > +++ b/systemd/nfs-idmapd.service
> > @@ -0,0 +1,9 @@
> > +[Unit]
> > +Description=NFSv4 ID-name mapping service
> Fedora has in the [Unit]:
> BindTo=nfs-server.service
> After=nfs-server.service
> I guess I thought they were needed at the time..

Yes, I saw that.  So I looked into it.
Apart from being a typo ("BindsTo" with an 's' is correct),
BindsTo has an extra effect if the target unit - nfs-server.service in this
case - disappears unexpectedly.  e.g. if the process dies.
However nfs-server.service doesn't have a process which can die.  Rather it
starts some kernel threads.  I'm pretty sure systemd won't be able to link
those threads with nfs-server.service.
So nfs-server.service can never "die", so "BindsTo" add nothing useful.

"After" just seems wrong.  The moment nfsd is started a request could come in
which could require an idmap lookup, so nfs-idmapd should already be there.

> 
> > +
> > +[Service]
> > +EnvironmentFile=-/run/sysconfig/nfs-utils
> > +ExecStartPre=-/usr/lib/systemd/scritps/nfs-utils_env.sh
> > +
> > +Type=forking
> > +ExecStart=/usr/sbin/rpc.idmapd $RPCIDMAPDARGS
> 
> 
> > diff --git a/systemd/nfs-mountd.service b/systemd/nfs-mountd.service
> > new file mode 100644
> > index 000000000000..92e05ca309ee
> > --- /dev/null
> > +++ b/systemd/nfs-mountd.service
> > @@ -0,0 +1,13 @@
> > +[Unit]
> > +Description=NFS Mount Daemon
> > +Requires=proc-fs-nfsd.mount
> > +After=proc-fs-nfsd.mount
> > +After=network.target
> > +PartOf=nfs-server.service
> > +
> > +[Service]
> > +EnvironmentFile=-/run/sysconfig/nfs-utils
> > +ExecStartPre=-/usr/lib/systemd/scritps/nfs-utils_env.sh
> How does this script know who is calling it and what it
> has to do? 

Its task is to read some config file like /etc/sysconfig/nfs and determine
what the command line arguments for each nfs program should be and write out
 FOO_ARGS=
lines to /run/sysconfig/nfs-utils for each FOO which is an nfs program.
I have since thought this would work better as a separate unit which creates
the nfs-utils env file once, and have all the other units have Wants/After
dependencies.


> 
> > +
> > +Type=forking
> > +ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS
> 
> 
> > diff --git a/systemd/nfs-secure.target b/systemd/nfs-secure.target
> > new file mode 100644
> > index 000000000000..0127fdb07dbd
> > --- /dev/null
> > +++ b/systemd/nfs-secure.target
> > @@ -0,0 +1,8 @@
> > +[Unit]
> > +Description=Secure NFS client/server services
> > +# If this target is enabled, then rpc.gssd and rpc.svcgssd will be started
> > +# as required.  If it is not enabled they won't.
> So the "Requisite=nfs-secure.target" in rpc-gssd.service cause the 
> daemon to started?

This is a bit subtle.
The "Requisite=nfs-secure.target" in rpc-gssd.service means that service is
only allowed to start if this unit (nfs-secure.target) is already started.
So if nfs-secure.target is enabled, then rpc-gssd.service will start when
anything Wants it.  If nfs-secure.target is not enabled, then
rpc-gssd.service will not start, even if something Wants it.

So enabling nfs-secure.target doesn't start anything, just allows a couple of
things to start if they are Wanted.


> 
> > +
> > +[Install]
> > +WantedBy=remote-fs.target
> > +WantedBy=multi-user.target
> > \ No newline at end of file
> 
> 
> > diff --git a/systemd/nfs-server.service b/systemd/nfs-server.service
> > new file mode 100644
> > index 000000000000..9812866c66aa
> > --- /dev/null
> > +++ b/systemd/nfs-server.service
> > @@ -0,0 +1,24 @@
> > +[Unit]
> > +Description=NFS server
> > +DefaultDependencies=no
> > +Requires= network.target proc-fs-nfsd.mount rpcbind.target
> > +PartOf=nfs-server.target
> > +
> > +After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
> > +After= nfs-idmapd.service rpc-statd.service
> > +After= rpc-gssd.service rpc-svcgssd.service
> > +Before= rpc-statd-notify.service
> > +
> > +[Service]
> > +EnvironmentFile=-/run/sysconfig/nfs-utils
> > +ExecStartPre=-/usr/lib/systemd/scritps/nfs-utils_env.sh
> > +
> > +Type=oneshot
> > +RemainAfterExit=yes
> > +ExecStartPre=/usr/sbin/exportfs -r
> > +ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
> Having ExecStartPre and ExecStartPost scripts gives
> us a place to do some pre and post server configuration.
> 
> Granted it has not been needed in a while but I'm thinking
> it could come in handing... 

systemd supports so-called "dropins".  If a distro wants to add an
ExecStartPre for some distro-specific reason they can put it in

 /etc/systemd/system/nfs-server.service.d/04-Fedora-hacks.conf

and it will be understood by systemd to be part of this service.
Of course if they were not distro-specific reasons we would include them in
the upstream package :-)

I note that the Fedora nfs-server.service has a preconfig
 echo "$NFSD_V4_GRACE" > /proc/fs/nfsd/nfsv4gracetime

and a postconfig
        /sbin/modprobe svcrdma
        echo "rdma $RDMA_PORT" > /proc/fs/nfsd/portlist

These seem reasonably general, though I'm wondering why the RDMA port is set
in postconfig rather than preconfig.
Do you know why that is?

> 
> > +ExecStop=/usr/sbin/rpc.nfsd 0
> > +ExecStopPost=/usr/sbin/exportfs -au
> > +ExecStopPost=/usr/sbin/exportfs -f
> > +
> > +ExecReload=/usr/sbin/exportfs -r
> 
> 
> > diff --git a/systemd/nfs-server.target b/systemd/nfs-server.target
> > new file mode 100644
> > index 000000000000..a3e629f022a9
> > --- /dev/null
> > +++ b/systemd/nfs-server.target
> > @@ -0,0 +1,8 @@
> > +[Unit]
> > +Description=NFS server services
> > +Requires=nfs-server.service nfs-mountd.service
> > +Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service
> > +Wants=rpc-statd-notify.service
> > +
> > +[Install]
> > +WantedBy=multi-user.target
> 
> 
> > diff --git a/systemd/proc-fs-nfsd.mount b/systemd/proc-fs-nfsd.mount
> > new file mode 100644
> > index 000000000000..f44d52f3d67b
> > --- /dev/null
> > +++ b/systemd/proc-fs-nfsd.mount
> > @@ -0,0 +1,8 @@
> > +[Unit]
> > +Description=NFSD configuration filesystem
> > +DefaultDependencies=no
> > +
> > +[Mount]
> > +What=nfsd
> > +Where=/proc/fs/nfsd
> > +Type=nfsd
> 
> 
> > diff --git a/systemd/rpc-gssd.service b/systemd/rpc-gssd.service
> > new file mode 100644
> > index 000000000000..f0fef007d480
> > --- /dev/null
> > +++ b/systemd/rpc-gssd.service
> > @@ -0,0 +1,14 @@
> > +[Unit]
> > +Description=RPC security service for NFS client and server
> > +Requires=var-lib-nfs-rpc_pipefs.mount
> > +After=var-lib-nfs-rpc_pipefs.mount
> > +
> > +Requisite=nfs-secure.target
> > +After=nfs-secure.target
> > +
> > +[Service]
> > +EnvironmentFile=-/run/sysconfig/nfs-utils
> > +ExecStartPre=-/usr/lib/systemd/scritps/nfs-utils_env.sh
> > +
> > +Type=forking
> > +ExecStart=/usr/sbin/rpc.gssd $GSSDARGS
> 
> 
> > diff --git a/systemd/rpc-statd-notify.service b/systemd/rpc-statd-notify.service
> > new file mode 100644
> > index 000000000000..9d972fc7753a
> > --- /dev/null
> > +++ b/systemd/rpc-statd-notify.service
> > @@ -0,0 +1,17 @@
> > +[Unit]
> > +Description=Notify NFS peers of a restart
> > +DefaultDependencies=no
> > +Requires=network-online.target
> > +After=network-online.target nss-lookup.target
> > +
> > +# if we run an nfs server, it needs to be running before we
> > +# tell clients that it has restarted.
> > +After=nfs-server.service
> > +
> > +[Service]
> > +EnvironmentFile=-/run/sysconfig/nfs-utils
> > +ExecStartPre=/usr/lib/systemd/scritps/nfs-utils_env.sh
> > +
> > +Type=oneshot
> > +RemainAfterExit=yes
> > +ExecStart=-/usr/sbin/sm-notify -d $SMNOTIFYARGS
> 
> 
> > diff --git a/systemd/rpc-statd.service b/systemd/rpc-statd.service
> > new file mode 100644
> > index 000000000000..04962e542fbc
> > --- /dev/null
> > +++ b/systemd/rpc-statd.service
> > @@ -0,0 +1,12 @@
> > +[Unit]
> > +Description=NFS status monitor for NFSv2/3 locking.
> > +DefaultDependencies=no
> > +Requires=nss-lookup.target rpcbind.target
> > +After=network.target nss-lookup.target rpcbind.target
> > +
> > +[Service]
> > +EnvironmentFile=-/run/sysconfig/nfs-utils
> > +ExecStartPre=-/usr/lib/systemd/scritps/nfs-utils_env.sh
> > +
> > +Type=forking
> > +ExecStart=/usr/sbin/rpc.statd --no-notify $STATDARGS
> 
> 
> > diff --git a/systemd/rpc-svcgssd.service b/systemd/rpc-svcgssd.service
> > new file mode 100644
> > index 000000000000..f024d40a8f41
> > --- /dev/null
> > +++ b/systemd/rpc-svcgssd.service
> > @@ -0,0 +1,15 @@
> > +[Unit]
> > +Description=RPC security service for NFS server
> > +Requires=var-lib-nfs-rpc_pipefs.mount
> > +After=var-lib-nfs-rpc_pipefs.mount
> > +PartOf=nfs-server.service
> > +
> > +Requisite=nfs-secure.target
> > +After=nfs-secure.target
> > +
> > +[Service]
> > +EnvironmentFile=-/run/sysconfig/nfs-utils
> > +ExecStartPre=-/usr/lib/systemd/scritps/nfs-utils_env.sh
> > +
> > +Type=forking
> > +ExecStart=/usr/sbin/rpc.svcgssd $SVCGSSDARGS
> 
> 
> > diff --git a/systemd/var-lib-nfs-rpc_pipefs.mount b/systemd/var-lib-nfs-rpc_pipefs.mount
> > new file mode 100644
> > index 000000000000..cd614cf49f00
> > --- /dev/null
> > +++ b/systemd/var-lib-nfs-rpc_pipefs.mount
> > @@ -0,0 +1,8 @@
> > +[Unit]
> > +Description=RPC Pipe File System
> > +DefaultDependencies=no
> > +
> > +[Mount]
> > +What=sunrpc
> > +Where=/var/lib/nfs/rpc_pipefs
> > +Type=rpc_pipefs
> 
> 
> > diff --git a/utils/statd/start-statd b/utils/statd/start-statd
> > index 1b345a547932..cde3583238e3 100644
> > --- a/utils/statd/start-statd
> > +++ b/utils/statd/start-statd
> > @@ -5,5 +5,8 @@
> >  # It should run statd with whatever flags are apropriate for this
> >  # site.
> >  PATH=/sbin:/usr/sbin
> > -exec rpc.statd --no-notify
> > -
> > +if systemctl start statd.service
> > +then :
> > +else
> > +    exec rpc.statd --no-notify
> > +fi
> > 
> 
> How does lockd get started and configured?

Lockd is automatically started by the kernel (nfs and nfsd) when needed.
I noticed that Fedora had some scripts to optionally set the port numbers
that lockd would use.  I left that out because I hoped we could find a better
way.

If lockd is compiled in to the kernel, then I think fs.nfs.nlm_XXXport should
be set via /etc/sysctl.conf.
If it is a module, then the module parameters should be configured via
an /etc/modprobe.d/lockd.conf file.
I wish this was unified somehow (should modprobe call sysctl??) but it isn't.

I appreciate that in neither case is it easy to get values out of a
'sysconfig' file in to the right place.

I cannot really think of a better solution than an 'nfs-lock' service which
sets these values,  so I'll probably add it.
> 
> Overall it looks reasonable... But this is a change
> of ABI... so it will be painful... :-(

What exactly is changing that will be painful?  I would certainly like to
arrange the unit files to minimise your pain as much as possible, as long as
it doesn't compromise effectiveness.

Thanks for your review!

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]