Re: Ceph-deploy refuses to create a named daemon

[Mark Kirkwood <mark.kirkwood@catalyst.net.nz>]

Yeah, makes sense.

Cheers

Mark
On 08/05/15 03:04, Travis Rhoden wrote:
> Hello again Mark,
>
> Thanks again for pointing this out. After some discussion, we would
> like to keep the 'client.rgw.*' naming convention, and we do need to
> update ceph-deploy to make sure people do not shoot themselves in the
> foot with this.
>
> It will still be totally fine to have custom names, but more along the
> lines of client.rgw.foo, client.rgw.bar, etc.
>
> I've filed http://tracker.ceph.com/issues/11561 to track this.
>
>   - Travis
>
> On Wed, Apr 22, 2015 at 9:24 AM, Travis Rhoden <trhoden@gmail.com> wrote:
>> Hi Mark,
>>
>> Thanks for the detective work!  I'll confirm whether the constraint on
>> the client name is really intended -- I believe that it is.  Once
>> confirmed, I completely agree that ceph-deploy should not allow you to
>> create rgw daemons with names that we know Ceph is going to reject.
>>
>>   - Travis
>>
>> On Wed, Apr 22, 2015 at 4:42 AM, Mark Kirkwood
>> <mark.kirkwood@catalyst.net.nz> wrote:
>>> On 13/04/15 15:33, Mark Kirkwood wrote:
>>>>
>>>> Hi,
>>>>
>>>> I've been experimenting with the new rgw creation in ceph-deploy, using
>>>> version 1.5.23 together with ceph 0.94 (-948-gd77de49).
>>>>
>>>> If simply run it without any args, then it works fine. e.g:
>>>>
>>>>
>>>> $ ceph-deploy  rgw create ceph1
>>>>
>>>>
>>>> However if I try to set a name for the daemon:
>>>>
>>>>
>>>> $ ceph-deploy  rgw create ceph1:radosgw.ceph1
>>>>
>>>>
>>>> then it fails to create and start a radosgw daemon at all. Poking
>>>> through the logs I see:
>>>>
>>>> $ tail ceph-client.radosgw.ceph1.log
>>>> 2015-04-13 14:13:20.513860 7f7e36c04840  0 librados:
>>>> client.radosgw.ceph1 authentication error (1) Operation not permitted
>>>>
>>>>
>>>> Not terribly informative. Looking at the audit log:
>>>>
>>>> $ tail ceph.audit.log
>>>> 015-04-13 15:10:37.880085 mon.0 192.168.122.21:6789/0 147 : audit [INF]
>>>> from='client.? 192.168.122.22:0/1004669' entity='client.bootstrap-rgw'
>>>> cmd=[{"prefix": "auth get-or-create", "entity": "client.radosgw.ceph2",
>>>> "caps": ["osd", "allow rwx", "mon", "allow rw"]}]:  access denied
>>>>
>>>>
>>>> Looking at what ceph-deploy is doing I see that the failing command is:
>>>>
>>>>
>>>> $ sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring
>>>> /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create
>>>> client.radosgw.ceph2 osd 'allow rwx' mon 'allow rw' -o
>>>> /var/lib/ceph/radosgw/ceph-radosgw.ceph2/keyring
>>>>
>>>>
>>>> This is failing whereas the equivalent command with radosgw replaced by
>>>> rgw works. Looking at what privileges client.bootstrap-rgw has I see:
>>>>
>>>> $ sudo ceph auth get client.bootstrap-rgw
>>>> [client.bootstrap-rgw]
>>>>       key = XXXXXXXXXXXXXXXXXX==
>>>>       caps mon = "allow profile bootstrap-rgw"
>>>>
>>>> Ah, so need to look at how the profile bootstrap-rgw is setup:
>>>>
>>>> $ more src/mon/MonCap.cc
>>>> ...
>>>>     if (profile == "bootstrap-rgw") {
>>>>       profile_grants.push_back(MonCapGrant("mon", MON_CAP_R));  // read
>>>> monmap
>>>>       profile_grants.push_back(MonCapGrant("osd", MON_CAP_R));  // read
>>>> osdmap
>>>>       profile_grants.push_back(MonCapGrant("mon getmap"));
>>>>       profile_grants.push_back(MonCapGrant("auth get-or-create"));  //
>>>> FIXME: this can expose other mds keys
>>>>       profile_grants.back().command_args["entity"] = StringConstraint("",
>>>> "client.rgw.");           <======
>>>>       profile_grants.back().command_args["caps_mon"] =
>>>> StringConstraint("allow rw", "");
>>>>       profile_grants.back().command_args["caps_osd"] =
>>>> StringConstraint("allow rwx", "");
>>>>     }
>>>> ...
>>>>
>>>> I see - the ["entity"] check requires the name to match "client.rgw.".
>>>> Double checking back on the deployed host (ceph1), I see that:
>>>>
>>>>
>>>> $ ceph --cluster ceph --name client.bootstrap-rgw --keyring
>>>> /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create
>>>> client.rgw.ceph3 osd 'allow rwx' mon 'allow rw'
>>>> [client.rgw.ceph3]
>>>>       key = AQCrNytVp9bqFBAAgTJVQDVQFJqiSkFqD68XsQ==
>>>> $ ceph --cluster ceph --name client.bootstrap-rgw --keyring
>>>> /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create
>>>> client.radosgw.ceph3 osd 'allow rwx' mon 'allow rw'
>>>> Error EACCES: access denied
>>>>
>>>> Ok so that is definitely the problem.
>>>>
>>>> Is this restriction intended? If so we really should make ceph-deploy
>>>> aware of this...as it is a bit of a foot gun (and not as easy as I made
>>>> it seem above to figure out *why* the deploy didn't work).
>>>>
>>>> Cheers
>>>>
>>>> Mark
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>> ping?
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>