systemd and ceph.conf and environment variables. (particularly mds and rgw)

[Owen Synge <osynge@suse.com>]

Dear all,

I have a simple idea that I think would be beneficial for ceph's
services. I would be curious to hear others opinions.

At the moment the only way (I know) to configure specific entities in
ceph such as a specific port for a mds or rgw is via ${CLUSTERNAME}.conf
which is usually set to ceph.conf. For brevity I will call this file
ceph.conf.

The current setup leads to complexities as ideally I want all ceph.conf
files to be identical across a cluster, and at the same time, I want to
add services such as rgw or mds without changing ceph.conf. I also want
to template a ceph.conf cluster.

If we could always get these settings via environment variables, it
would make setting up ceph with tools such as cluster/config management
tools such as puppet, chef, and salt easier.

So here is the diff of the systemd service files I am thinking of:

# diff -u  ceph-mds@.service /tmp/ceph-mds@.service
--- ceph-mds@.service   2016-03-01 20:48:05.000000000 -0500
+++ /tmp/ceph-mds@.service      2016-03-07 05:14:43.392000000 -0500
@@ -8,8 +8,8 @@
 LimitNOFILE=1048576
 LimitNPROC=1048576
 EnvironmentFile=-/etc/sysconfig/ceph
-Environment=CLUSTER=ceph
-ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id %i --setuser
ceph --setgroup ceph
+EnvironmentFile=-/var/lib/ceph/env/mds/%i
+ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} --id %i --setuser
ceph --setgroup ceph -m ${BIND_IPV4}:${BIND_PORT}
 ExecReload=/bin/kill -HUP $MAINPID
 PrivateDevices=yes
 ProtectHome=true

First you should note the extra EnvironmentFile line:

EnvironmentFile=-/var/lib/ceph/mds/systemd/%i

This means that if the mds service is started with parameter "param_one"
then the environment will be set with the content of

    /var/lib/ceph/env/mds/param_one

The "-" means don’t error if the above file does not exist.

Hence we could add service instance variables such as ${BIND_IPV4} and
${BIND_PORT} as well as important values like ${CLUSTER} to values like
ceph.

Having played with systemd, it seems that environment variables are
updated by the last assignment in systemd just like shell.

I have *not* found a good way to put in conditionals in such systemd
"EnvironmentFile" content. Hence it would be impossible to nicely
default command line parameters without using dirty tricks. Specifically
what we cant do in the systemd "EnvironmentFile" context:

	if [ X"${BIND_IPV4}${BIND_PORT}" != X ] ; then
	EXTRA_ARGS="-m ${BIND_IPV4}:${BIND_PORT}"
	fi

Hence I think we have to accept to integrate with systemd nicely we
should allow setting to be read from environment rather than only from
the command line.

What do people think?

Best wishes

Owen



--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html