understanding cephx

understanding cephx

[Michael Menge]

[-- Attachment #1: Type: text/plain, Size: 4394 bytes --]

Hi,

before I get to my questions, I want to thank for the good work done
with ceph. I learned about ceph in an Admin-Magazin article [1]
and was supprised how easy it was to setup ceph by following the
article. Trying new software and not hitting any error/warning or
other problems is a very rare incident and I was verry impressed
by the easy installation and configuration.

Later on I had some smaler problems as i tried to increase the number
of mon, ods an by adding an standby mds. But i managed to figure it
out using manpages and the web.

Now I have a problem that I don't know how to fix.

First some informations about my setup
ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)

ceph.conf
------------
[global]
         ; Enable authentication between hosts within the cluster.
         auth supported = cephx
         keyring = /etc/ceph/$name.keyring

[mon]

         mon data = /srv/mon.$id

[mds]

[osd]
         osd data = /srv/osd.$id
         osd journal = /srv/osd.$id.journal
         osd journal size = 1000

[mon.b]
         host = hpb020102
         mon addr = 10.23.3.2:6789

[mon.c]
         host = hpb020103
         mon addr = 10.23.3.3:6789


[mon.d]
         host = hpb020104
         mon addr = 10.23.3.4:6789

[mon.e]
         host = hpb020105
         mon addr = 10.23.3.5:6789

[mon.f]
         host = hpb020106
         mon addr = 10.23.3.6:6789
[osd.2]
         host = hpb020102

[osd.3]
         host = hpb020103

[osd.4]
         host = hpb020104

[osd.5]
         host = hpb020105

[osd.6]
         host = hpb020106

[mds.a]
         host = hpb020104

[mds.b]
         host = hpb020105
         mds standby replay = true
------------
/srv/osd.* are on xfs  partition


Befor my holiday I found logs that indicated that there might
be a problem with one of my mds which is still present

     2013-01-14 15:32:41.943304   mds e515692: 1/1/1 up  
{0=a=up:active}, 1 up:standby-replay, 5 up:oneshot-replay(laggy or  
crashed)

I tried to increase the log-level and get some debug infos.
After my holiday i found that the ceph-logs mostly the mon log had
filled my / filesystem. First I thougth that the debugging was still
active but at a closer look, I found that somehow the mon. key could
not be found by mon.e

2013-01-14 15:44:52.007632 7fad1e728700  0 mon.e@3(probing) e3   
couldn't get secret for mon service
2013-01-14 15:44:52.007655 7fad17ee9700  0 mon.e@3(probing) e3   
couldn't get secret for mon service
2013-01-14 15:44:52.007659 7fad1e728700  0 mon.e@3(probing) e3 no  
installed auth entries!
2013-01-14 15:44:52.007662 7fad17ee9700  0 mon.e@3(probing) e3 no  
installed auth entries!
2013-01-14 15:44:52.007860 7fad17ee9700  0 -- 10.23.3.5:6789/0 >>  
10.23.3.3:6789/0 pipe(0x8e7190 sd=19 pgs=0 cs=0 l=0
).connect got BADAUTHORIZER
2013-01-14 15:44:52.007860 7fad1e728700  0 -- 10.23.3.5:6789/0 >>  
10.23.3.2:6789/0 pipe(0x8e6870 sd=18 pgs=0 cs=0 l=0
).connect got BADAUTHORIZER

So i guess, by trying to get some more informations I somehow
manged to delete the mon. key. I was unable the retieve the history
because of the full filesystem.

So I tried to use "ceph auth" and ceph-authtool to (re-)add the mon. key
but only managed that mon.d is now too unable the authenticate.

Sofar I know that I don't understand how cephx is working.

"ceph auth list" shows the same key for mon. on all servers.
But as it takes longer on hpb020104 and hpb020105 I guess it
will contact some other mon servers as mon.d and mon.e are out
of quorum.

How can i get informations about the mon. key for mon.d and mon.e
if they are not running / out of quorum?
How can I add/change the mon. key?

"/etc/ceph/" has keyrings for admin client.admin mds.* ods.* but none
for mon. or mon.*
Is this correct?


Best regards

    Michael Menge





[1]  
http://www.admin-magazin.de/Das-Heft/2012/03/Der-RADOS-Objectstore-und-Ceph-Teil-1/%28language%29/ger-DE


--------------------------------------------------------------------------------

M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

[-- Attachment #2: S/MIME Signatur --]
[-- Type: application/pkcs7-signature, Size: 5267 bytes --]

Re: understanding cephx

[Michael Menge]

[-- Attachment #1: Type: text/plain, Size: 1172 bytes --]

Quoting Michael Menge <michael.menge@zdv.uni-tuebingen.de>:

>
> So i guess, by trying to get some more informations I somehow
> manged to delete the mon. key. I was unable the retieve the history
> because of the full filesystem.
>
> So I tried to use "ceph auth" and ceph-authtool to (re-)add the mon. key
> but only managed that mon.d is now too unable the authenticate.
>

I was able to insert the mon. key by receating the mon servers

ceph auth get mon. -o /tmp/monkey.new
ceph mon getmap -o /tmp/monmap.new
ceph-mon -i d --mkfs --monmap /tmp/monmap.new --keyring /tmp/monkey.new

Restarting the mon server didn't seem to have worked at first.
Restarting with --debug_mon 10 showed that I was too impatient
as the mon server had to update his mdsmap which took quiet some time

Regards

    Michael Menge

--------------------------------------------------------------------------------

M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

[-- Attachment #2: S/MIME Signatur --]
[-- Type: application/pkcs7-signature, Size: 5267 bytes --]

Re: understanding cephx

[Dan Mick]

Glad you got it working!  Sorry for missing your original message..

On 01/16/2013 01:47 AM, Michael Menge wrote:
> Quoting Michael Menge <michael.menge@zdv.uni-tuebingen.de>:
>
>>
>> So i guess, by trying to get some more informations I somehow
>> manged to delete the mon. key. I was unable the retieve the history
>> because of the full filesystem.
>>
>> So I tried to use "ceph auth" and ceph-authtool to (re-)add the mon. key
>> but only managed that mon.d is now too unable the authenticate.
>>
>
> I was able to insert the mon. key by receating the mon servers
>
> ceph auth get mon. -o /tmp/monkey.new
> ceph mon getmap -o /tmp/monmap.new
> ceph-mon -i d --mkfs --monmap /tmp/monmap.new --keyring /tmp/monkey.new
>
> Restarting the mon server didn't seem to have worked at first.
> Restarting with --debug_mon 10 showed that I was too impatient
> as the mon server had to update his mdsmap which took quiet some time
>
> Regards
>
>     Michael Menge
>
> --------------------------------------------------------------------------------
>
>
> M.Menge                                Tel.: (49) 7071/29-70316
> Universität Tübingen                   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung          mail:
> michael.menge@zdv.uni-tuebingen.de
> Wächterstraße 76
> 72074 Tübingen
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html