running daemons as user/group ceph

running daemons as user/group ceph

[Sage Weil]

I started to put together patches to create a ceph user and group.  
You can see them here:

	https://github.com/ceph/ceph/pull/4456

For now I just did the debian bits.  Hopefully someone more familiar with 
rpms can do the spec file?

I put in the UID Ken mentioned for Debian.  Has that been allocated yet?

On package install it chowns /var/lib/ceph to the ceph user.  I don't want 
to do a recursive chown because there may be an unbounded amount of data 
there (and a running daemon).  Instead the packages make themselves 
responsible for chowning /var/lib/ceph and the first level of subdirs.  
The mode is 750; I'm not sure it really matters what the ceph 
group can do here.

-- Logs --

The /var/log/ceph directory is trickier.  We also chown it, but I set the 
mode with

 chmod u=rwx,g=rxs,o= /var/log/ceph

The idea is to allow members of the ceph group to log to this directory.  
For example, qemu could be added to group ceph so that when it links 
librbd the client logs will show up here.  (Same goes for /var/run/ceph 
and the admin socket.)  But it also means that daemon logs become 
writeable by that user.  Maybe the log file mode should be 700 so that 
everyone can create files here but not modify others' logs?  Or is the 
idea of letting librados users log here just a bad idea?

On test case is radosgw, which currently logs to /var/log/radosgw but I 
would like to change to log to /var/log/ceph if we can for simplicity.  
It will run as the ceph user, though, whereas something like qemu would 
not.  What about an 'rbd' or 'ceph' CLI user?  Should that log to 
/var/log/ceph?  Only if the user is in the ceph group?

-- systemd --

Most of the daemons can just get the User=ceph and Group=cpeh lines in the 
unit files.  The OSD is tricky, though, since we want the prestart script 
to run as root so that it can chown the disk contents if necessary.  We 
have two options, I think:

1) run prestart and ceph-osd as root, and add a ceph daemon arg to drop 
privileges and setuid.

2) add a sudo rule so that the ceph user can run the chown command from 
prestart.  (This seems more dangerous.)

Thoughts?

sage

Re: running daemons as user/group ceph

[Danny Al-Gaaf]

Am 24.04.2015 um 19:37 schrieb Sage Weil:
[...]
> -- systemd --
> 
> Most of the daemons can just get the User=ceph and Group=cpeh lines in the 
> unit files.  The OSD is tricky, though, since we want the prestart script 
> to run as root so that it can chown the disk contents if necessary.  We 
> have two options, I think:
> 
> 1) run prestart and ceph-osd as root, and add a ceph daemon arg to drop 
> privileges and setuid.
> 
> 2) add a sudo rule so that the ceph user can run the chown command from 
> prestart.  (This seems more dangerous.)
> 
> Thoughts?

Do we need to change the start scripts for SysV init? Or is this
something we should ignore because the most distros will use systemd in
the future.

Danny

Re: running daemons as user/group ceph

[Sage Weil]

On Fri, 24 Apr 2015, Danny Al-Gaaf wrote:
> Am 24.04.2015 um 19:37 schrieb Sage Weil:
> [...]
> > -- systemd --
> > 
> > Most of the daemons can just get the User=ceph and Group=cpeh lines in the 
> > unit files.  The OSD is tricky, though, since we want the prestart script 
> > to run as root so that it can chown the disk contents if necessary.  We 
> > have two options, I think:
> > 
> > 1) run prestart and ceph-osd as root, and add a ceph daemon arg to drop 
> > privileges and setuid.
> > 
> > 2) add a sudo rule so that the ceph user can run the chown command from 
> > prestart.  (This seems more dangerous.)
> > 
> > Thoughts?
> 
> Do we need to change the start scripts for SysV init? Or is this
> something we should ignore because the most distros will use systemd in
> the future.

We could, but I wonder if not touching upstart or sysvinit will be an 
easy way to handle migration/compat issues.

One other thing Greg brought up today was that we should allow an admin to 
configure daemons to run as root if they want.  They can do that by 
editing the unit files; I'm not sure if we want to do something more 
friendly than that?  (FWIW I think this is basically what Leannart 
suggests.)

sage

Re: running daemons as user/group ceph

[Robert LeBlanc]

On Fri, Apr 24, 2015 at 2:52 PM, Sage Weil <sweil@redhat.com> wrote:
>
> We could, but I wonder if not touching upstart or sysvinit will be an
> easy way to handle migration/compat issues.
>
> One other thing Greg brought up today was that we should allow an admin to
> configure daemons to run as root if they want.  They can do that by
> editing the unit files; I'm not sure if we want to do something more
> friendly than that?  (FWIW I think this is basically what Leannart
> suggests.)

How about putting a user option in /etc/default/ceph or ceph.conf?

Re: running daemons as user/group ceph

[Danny Al-Gaaf]

Am 24.04.2015 um 22:52 schrieb Sage Weil:
> On Fri, 24 Apr 2015, Danny Al-Gaaf wrote:
>> Am 24.04.2015 um 19:37 schrieb Sage Weil:
>> [...]
>>> -- systemd --
>>>
>>> Most of the daemons can just get the User=ceph and Group=cpeh lines in the 
>>> unit files.  The OSD is tricky, though, since we want the prestart script 
>>> to run as root so that it can chown the disk contents if necessary.  We 
>>> have two options, I think:
>>>
>>> 1) run prestart and ceph-osd as root, and add a ceph daemon arg to drop 
>>> privileges and setuid.
>>>
>>> 2) add a sudo rule so that the ceph user can run the chown command from 
>>> prestart.  (This seems more dangerous.)
>>>
>>> Thoughts?
>>
>> Do we need to change the start scripts for SysV init? Or is this
>> something we should ignore because the most distros will use systemd in
>> the future.
> 
> We could, but I wonder if not touching upstart or sysvinit will be an 
> easy way to handle migration/compat issues.
> 
> One other thing Greg brought up today was that we should allow an admin to 
> configure daemons to run as root if they want.  They can do that by 
> editing the unit files; I'm not sure if we want to do something more 
> friendly than that?  (FWIW I think this is basically what Leannart 
> suggests.)

The alternative would be to drop the privileges within the code of the
daemons (as soon as possible) and use a config/cmdline option to check
if we want to start the daemons as root or under the ceph user.

Danny

Re: running daemons as user/group ceph

[Ken Dreyer]

On 04/24/2015 11:37 AM, Sage Weil wrote:
> -- Logs --
> 

One other thing in addition to the log directory is the socket directory
permissions (/var/run/ceph). The ceph UID will need to write there, right?

In newer distros with systemd, /var/run is on tmpfs so we use this
tmpfiles.d snippet to be sure the directory is there in /var/tmpfs after
every boot:

https://github.com/ceph/ceph/blob/master/systemd/ceph.tmpfiles.d

The snippet currently creates the directory as root-owned, and I imagine
we'd want to change that to the ceph UID instead?

> -- systemd --
> 
> Most of the daemons can just get the User=ceph and Group=cpeh lines in the 
> unit files.  The OSD is tricky, though, since we want the prestart script 
> to run as root so that it can chown the disk contents if necessary.  We 
> have two options, I think:
> 
> 1) run prestart and ceph-osd as root, and add a ceph daemon arg to drop 
> privileges and setuid.
> 
> 2) add a sudo rule so that the ceph user can run the chown command from 
> prestart.  (This seems more dangerous.)

I agree sudo sounds more dangerous, and it'll also be more complex to
implement in the packaging.

Would it be possible to use Apache's model, where it does the bare
minimum set of things it needs as root (binding to port 80, etc), and
then drops privileges thereafter?

If the OSD had this ability built-in, then it could run in minimal
environments like containers where sudo is not present, etc.

- Ken

Re: running daemons as user/group ceph

[Sage Weil]

On Fri, 24 Apr 2015, Ken Dreyer wrote:
> On 04/24/2015 11:37 AM, Sage Weil wrote:
> > -- Logs --
> > 
> 
> One other thing in addition to the log directory is the socket directory
> permissions (/var/run/ceph). The ceph UID will need to write there, right?
> 
> In newer distros with systemd, /var/run is on tmpfs so we use this
> tmpfiles.d snippet to be sure the directory is there in /var/tmpfs after
> every boot:
> 
> https://github.com/ceph/ceph/blob/master/systemd/ceph.tmpfiles.d
> 
> The snippet currently creates the directory as root-owned, and I imagine
> we'd want to change that to the ceph UID instead?

Yeah, I changed that file.  Again, not certain about the group permissions 
bit, though.

> > -- systemd --
> > 
> > Most of the daemons can just get the User=ceph and Group=cpeh lines in the 
> > unit files.  The OSD is tricky, though, since we want the prestart script 
> > to run as root so that it can chown the disk contents if necessary.  We 
> > have two options, I think:
> > 
> > 1) run prestart and ceph-osd as root, and add a ceph daemon arg to drop 
> > privileges and setuid.
> > 
> > 2) add a sudo rule so that the ceph user can run the chown command from 
> > prestart.  (This seems more dangerous.)
> 
> I agree sudo sounds more dangerous, and it'll also be more complex to
> implement in the packaging.
> 
> Would it be possible to use Apache's model, where it does the bare
> minimum set of things it needs as root (binding to port 80, etc), and
> then drops privileges thereafter?
> 
> If the OSD had this ability built-in, then it could run in minimal
> environments like containers where sudo is not present, etc.

Yeah. And in this case, it's only the ceph-osd-prestart.sh script (run by 
systemd) that needs root; the ceph-osd can drop privileges immediately 
upon starting.

sage

Re: [Ceph-maintainers] running daemons as user/group ceph

[Sage Weil]

On Fri, 24 Apr 2015, Sage Weil wrote:
> > > -- systemd --
> > > 
> > > Most of the daemons can just get the User=ceph and Group=cpeh lines in the 
> > > unit files.  The OSD is tricky, though, since we want the prestart script 
> > > to run as root so that it can chown the disk contents if necessary.  We 
> > > have two options, I think:
> > > 
> > > 1) run prestart and ceph-osd as root, and add a ceph daemon arg to drop 
> > > privileges and setuid.
> > > 
> > > 2) add a sudo rule so that the ceph user can run the chown command from 
> > > prestart.  (This seems more dangerous.)
> > 
> > I agree sudo sounds more dangerous, and it'll also be more complex to
> > implement in the packaging.
> > 
> > Would it be possible to use Apache's model, where it does the bare
> > minimum set of things it needs as root (binding to port 80, etc), and
> > then drops privileges thereafter?
> > 
> > If the OSD had this ability built-in, then it could run in minimal
> > environments like containers where sudo is not present, etc.
> 
> Yeah. And in this case, it's only the ceph-osd-prestart.sh script (run by 
> systemd) that needs root; the ceph-osd can drop privileges immediately 
> upon starting.

Pushed a patch that lets daemons drop privs (setuid and setgid) and 
updated the ceph-osd unit file accordingly.

Also added a /etc/security/limits.d/ceph file so that we can set the 
nofile ulimit in one place where people expect it and not in our configs 
or the systemd/upstart configs.

sage

Re: running daemons as user/group ceph

[Ken Dreyer]

On 04/24/2015 03:13 PM, Sage Weil wrote:
> On Fri, 24 Apr 2015, Ken Dreyer wrote:
>> On 04/24/2015 11:37 AM, Sage Weil wrote:
>>> -- Logs --
>>>
>>
>> One other thing in addition to the log directory is the socket directory
>> permissions (/var/run/ceph). The ceph UID will need to write there, right?
>>
>> In newer distros with systemd, /var/run is on tmpfs so we use this
>> tmpfiles.d snippet to be sure the directory is there in /var/tmpfs after
>> every boot:
>>
>> https://github.com/ceph/ceph/blob/master/systemd/ceph.tmpfiles.d
>>
>> The snippet currently creates the directory as root-owned, and I imagine
>> we'd want to change that to the ceph UID instead?
> 
> Yeah, I changed that file.  Again, not certain about the group permissions 
> bit, though.

Oh, my bad, sorry I missed that part of the pull request! Looks good to me.

- Ken

Re: running daemons as user/group ceph

[Sage Weil]

On Fri, 24 Apr 2015, Ken Dreyer wrote:
> Oh, my bad, sorry I missed that part of the pull request! Looks good to me.

Still missing ceph.spec file changes, if you're up for it.  :)

I'm working on the chown piece, but I'm not sure what to do about the 
journal device (if it is a device).  We can either

 - add ceph to the 'disk' group.  This means it can scribble on any device 
in the system, which seems less than ideal.

 - chown the journal device to ceph when the daemon is started.  This 
means the user can only scribble on other ceph devices, which seems about 
right.  But I'm not sure what the implications of chowning random devices 
like /dev/sdc2 to ceph are.

 - do something tricky where we open the block device on startup before 
doing the setuid, and reuse that file handle later.  This likely screws up 
all kinds layering because we want the privs to drop very eary (before we 
start opening log files, for example) and only the journal code knows what 
mode to open the file as.  Any solution here will be kind of kludgey, I 
think.

?

sage

Re: running daemons as user/group ceph

[Danny Al-Gaaf]

Am 25.04.2015 um 01:34 schrieb Sage Weil:
> On Fri, 24 Apr 2015, Ken Dreyer wrote:
>> Oh, my bad, sorry I missed that part of the pull request! Looks good to me.
> 
> Still missing ceph.spec file changes, if you're up for it.  :)

I can change the spec files, but do we have finally some user/group IDs
for RHEL/Fedora and SLE/openSUSE?

> I'm working on the chown piece, but I'm not sure what to do about the 
> journal device (if it is a device).  We can either
> 
>  - add ceph to the 'disk' group.  This means it can scribble on any device 
> in the system, which seems less than ideal.

I wouldn't do that. Especially from the security perspective.

>  - chown the journal device to ceph when the daemon is started.  This 
> means the user can only scribble on other ceph devices, which seems about 
> right.  But I'm not sure what the implications of chowning random devices 
> like /dev/sdc2 to ceph are.

Not sure if this is what we want or that this is the "usual" way to
handle these issues.

>  - do something tricky where we open the block device on startup before 
> doing the setuid, and reuse that file handle later.  This likely screws up 
> all kinds layering because we want the privs to drop very eary (before we 
> start opening log files, for example) and only the journal code knows what 
> mode to open the file as.  Any solution here will be kind of kludgey, I 
> think.

The log files are not the issue here I guess since you can set the
correct permissions outside of the daemons. If an OSD need to open a
device it would be appropriate to open the device as root before
dropping the privileges. The code shouldn't be that complicated we did
such things e.g. in hald before.

Another issue: there is the --enable-root-make-check configure option. I
didn't take a look at the code yet, but does anybody know if these tests
refer to something that needs root privileges within the daemons or is
it due to testing something (like e.g. mount rbd's)?

Danny

Re: running daemons as user/group ceph

[Sage Weil]

On Sat, 25 Apr 2015, Danny Al-Gaaf wrote:
> Am 25.04.2015 um 01:34 schrieb Sage Weil:
> > On Fri, 24 Apr 2015, Ken Dreyer wrote:
> >> Oh, my bad, sorry I missed that part of the pull request! Looks good to me.
> > 
> > Still missing ceph.spec file changes, if you're up for it.  :)
> 
> I can change the spec files, but do we have finally some user/group IDs
> for RHEL/Fedora and SLE/openSUSE?

Not yet, but I think the specfile should have a generic adduser (or 
whatever) that just allocates a uid for distros where the uid isn't 
preallocated; using the fixed uid will be a conditional special case for 
those distros where it is fixed.

> > I'm working on the chown piece, but I'm not sure what to do about the 
> > journal device (if it is a device).  We can either
> > 
> >  - add ceph to the 'disk' group.  This means it can scribble on any device 
> > in the system, which seems less than ideal.
> 
> I wouldn't do that. Especially from the security perspective.

Yeah

> >  - chown the journal device to ceph when the daemon is started.  This 
> > means the user can only scribble on other ceph devices, which seems about 
> > right.  But I'm not sure what the implications of chowning random devices 
> > like /dev/sdc2 to ceph are.
> 
> Not sure if this is what we want or that this is the "usual" way to
> handle these issues.

It's certainly the simplest, it if passes muster...

> >  - do something tricky where we open the block device on startup before 
> > doing the setuid, and reuse that file handle later.  This likely screws up 
> > all kinds layering because we want the privs to drop very eary (before we 
> > start opening log files, for example) and only the journal code knows what 
> > mode to open the file as.  Any solution here will be kind of kludgey, I 
> > think.
> 
> The log files are not the issue here I guess since you can set the
> correct permissions outside of the daemons. If an OSD need to open a
> device it would be appropriate to open the device as root before
> dropping the privileges. The code shouldn't be that complicated we did
> such things e.g. in hald before.

I'm just worried that if we open(O_CREAT) the log files before dropping 
privs then they'll be owned by root instead of ceph.  And that any other 
random stuff that happens in the relatively long gap between start and 
opening the FileStore's journal will have elevated privs and create files 
(pid file, admin socket file in /var/run/ceph, etc.).  And special casing 
moving the journal open to the beginning will be tricky because the 
ObjectStore is pluggable, and because we have to do the FileStore init 
(including thread creation etc) after the fork.

> Another issue: there is the --enable-root-make-check configure option. I 
> didn't take a look at the code yet, but does anybody know if these tests 
> refer to something that needs root privileges within the daemons or is 
> it due to testing something (like e.g. mount rbd's)?

I suspect it's the ceph-disk tests?

sage