normalizing radosgw

normalizing radosgw

[Sage Weil]

Several things are different/annoying with radosgw than with other Ceph 
daemons:

- binary/package are named 'radosgw' instead of 'ceph-rgw'.

This is cosmetic, but it also makes it fit less well into the 
new /var/lib/ceph/* view of things.

- default log location is /var/log/radosgw/$cluster-$name.log instead of 
/var/lib/ceph/$cluster-$rgw.$name.log (or similar).

- rgw_data default is /var/lib/ceph/radosgw instead of /var/lib/ceph/rgw

(not sure if 3 letters for consistency is better?)

- rgw usually authenticates as a client.something user, which means if you 
do use more standard rgw log names, then you get 
/var/log/ceph/client.something.$pid.$uniqueid.log.  There is a loose 
convention that 'something' is 'rgw.hostname' (i.e., client.rgw.hostname).

- radosgw has it's own separate sysvinit script that enumerates daemons 
from /etc/ceph/ceph.conf sections that start with client.radosgw.*

- radosgw upstart script is called radosgw (not ceph-rgw) and enumerates 
daemons from /var/lib/ceph/radosgw/*

(totally different than sysvinit!)

- radosgw instances usually need some stuff in ceph.conf to make them 
behave properly, which means they need a section in the shared 
/etc/ceph/ceph.conf.  more work for admins or config management systems.

- on rpm-based systems we have a separate sysvinit script that is 
slightly different (mostly because the username is different, apache 
instead of www-data).

---

There is enough wrong here and little enough convention that my 
proposal is to essentially start fresh:

- rename package ceph-rgw.  obsoletes/replaces radosgw.  

- merge rgw start/stop into standard sysvinit script (just another 
daemon type?).

- normalize upstart start/stop (just rename radosgw-* to ceph-rgw-*, 
basically).

- make upstart use /var/lib/ceph/rgw instead of /var/lib/ceph/radosgw (do 
this automagically on upgrade?  that will at least avoid the upgrade pain 
for ubuntu users)

- create new systemd start/stop that are correct the first time 
around.

- move log file to /var/log/ceph


The part I'm not sure about is how to handle config.  I would really like 
the ability to put per-daemon config stuff in /var/lib/ceph/rgw/foo/conf 
or config, but there is no 'include file' function in our ini-file parser 
(or in other typical ini-file implementations that i can find) and I don't 
like the idea the per-daemon config would completely supplant the 
/etc/ceph/ one.

Basically, I'd like to get to a point where we can programatically deploy 
an rgw without any wonky code that edits the ceph.conf.  Perhaps the way 
to accomplish this is to have the admin create a generic client.rgw user 
and generic [client.rgw] section in the global ceph.conf?

If there were an include file function, we could make the per-daemon 
config file something like

 include /etc/ceph/$cluster.conf
 [client.rgw.hostname]
 rgw foo = bar
 rgw baz = blip

or whatever.

Thoughts?  Suggestions?

sage

Re: [ceph-users] normalizing radosgw

[Yehuda Sadeh]

On Sat, Dec 6, 2014 at 10:39 AM, Sage Weil <sweil@redhat.com> wrote:
> Several things are different/annoying with radosgw than with other Ceph
> daemons:
>
> - binary/package are named 'radosgw' instead of 'ceph-rgw'.
>
> This is cosmetic, but it also makes it fit less well into the
> new /var/lib/ceph/* view of things.
>
> - default log location is /var/log/radosgw/$cluster-$name.log instead of
> /var/lib/ceph/$cluster-$rgw.$name.log (or similar).
>
> - rgw_data default is /var/lib/ceph/radosgw instead of /var/lib/ceph/rgw
>
> (not sure if 3 letters for consistency is better?)
>
> - rgw usually authenticates as a client.something user, which means if you
> do use more standard rgw log names, then you get
> /var/log/ceph/client.something.$pid.$uniqueid.log.  There is a loose
> convention that 'something' is 'rgw.hostname' (i.e., client.rgw.hostname).

Maybe we can make it so that unless user name starts with mon, osd, or
mds, it's going to be treated as 'client'? Or something along these
lines.

>
> - radosgw has it's own separate sysvinit script that enumerates daemons
> from /etc/ceph/ceph.conf sections that start with client.radosgw.*
>
> - radosgw upstart script is called radosgw (not ceph-rgw) and enumerates
> daemons from /var/lib/ceph/radosgw/*
>
> (totally different than sysvinit!)
>
> - radosgw instances usually need some stuff in ceph.conf to make them
> behave properly, which means they need a section in the shared
> /etc/ceph/ceph.conf.  more work for admins or config management systems.
>
> - on rpm-based systems we have a separate sysvinit script that is
> slightly different (mostly because the username is different, apache
> instead of www-data).
>
> ---
>
> There is enough wrong here and little enough convention that my
> proposal is to essentially start fresh:
>
> - rename package ceph-rgw.  obsoletes/replaces radosgw.
>
> - merge rgw start/stop into standard sysvinit script (just another
> daemon type?).
>
> - normalize upstart start/stop (just rename radosgw-* to ceph-rgw-*,
> basically).
>
> - make upstart use /var/lib/ceph/rgw instead of /var/lib/ceph/radosgw (do
> this automagically on upgrade?  that will at least avoid the upgrade pain
> for ubuntu users)
>
> - create new systemd start/stop that are correct the first time
> around.
>
> - move log file to /var/log/ceph
>
>
> The part I'm not sure about is how to handle config.  I would really like
> the ability to put per-daemon config stuff in /var/lib/ceph/rgw/foo/conf
> or config, but there is no 'include file' function in our ini-file parser
> (or in other typical ini-file implementations that i can find) and I don't

The samba configuration files, which our ceph conf was heavily
influenced by does have that capability. We also used to be able to
do:

   include = <other include file>

but sadly that was taken out during some cleanup years ago. I don't
think it's going to be too complicated to revive this functionality.

> like the idea the per-daemon config would completely supplant the
> /etc/ceph/ one.
>
> Basically, I'd like to get to a point where we can programatically deploy
> an rgw without any wonky code that edits the ceph.conf.  Perhaps the way
> to accomplish this is to have the admin create a generic client.rgw user
> and generic [client.rgw] section in the global ceph.conf?
>
> If there were an include file function, we could make the per-daemon
> config file something like
>
>  include /etc/ceph/$cluster.conf
>  [client.rgw.hostname]
>  rgw foo = bar
>  rgw baz = blip
>
> or whatever.
>
> Thoughts?  Suggestions?
>
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: normalizing radosgw

[Mark Kirkwood]

On 07/12/14 07:39, Sage Weil wrote:
>
> Thoughts?  Suggestions?
>


Would kit make sense to include radosgw-agent package in this 
normalization too?

Regards

Mark

Re: normalizing radosgw

[Anthony Alba]

[-- Attachment #1.1: Type: text/plain, Size: 412 bytes --]

Suggestion: can we look at normalizing custom cluster names as well for
RHEL-like systems?

/etc/init.d/ceph: how do you pass --cluster myname at system startup?
systemd service file uses EnvironmentFile.

What about
. /etc/sysconfig/ceph #near the top

and /etc/sysconfig/ceph contains
cluster=myname

For ceph-radosgw even this does not work.
You need
export CEPH_CONF=/etc/ceph/myname.conf

as well.

Anthony

[-- Attachment #1.2: Type: text/html, Size: 566 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] normalizing radosgw

[Abhishek L]

[-- Attachment #1: Type: text/plain, Size: 221 bytes --]


Sage Weil writes:
[..]
> Thoughts?  Suggestions?
>
[..]
Suggestion:
radosgw should handle injectargs like other ceph clients do?

This is not a major annoyance, but it would be nice to have.


-- 
Abhishek

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]