ceph-create-keys hung

ceph-create-keys hung

[Abhay Sachan]

[-- Attachment #1.1: Type: text/plain, Size: 423 bytes --]

Hi All,
I have tried setting up a ceph cluster with 3 nodes (3 monitors). I am
using RHEL 6.4 as OS with dumpling(0.67.3) release. Ceph cluster creation
(using ceph-deploy as well as mkcephfs), ceph-creates-keys doesn't return
on any of the servers. Whereas, if I create a cluster with only 1 node (1
monitor), key creation goes through. Has anybody seen this problem or any
ideas what I might be missing??

Regards,
Abhay

[-- Attachment #1.2: Type: text/html, Size: 502 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph-create-keys hung

[Joao Eduardo Luis]

On 10/03/2013 02:44 PM, Abhay Sachan wrote:
> Hi All,
> I have tried setting up a ceph cluster with 3 nodes (3 monitors). I am
> using RHEL 6.4 as OS with dumpling(0.67.3) release. Ceph cluster
> creation (using ceph-deploy as well as mkcephfs), ceph-creates-keys
> doesn't return on any of the servers. Whereas, if I create a cluster
> with only 1 node (1 monitor), key creation goes through. Has anybody
> seen this problem or any ideas what I might be missing??
>
> Regards,
> Abhay

Those symptoms tell me that your monitors are not forming quorum. 
'ceph-create-keys' needs the monitors to first establish a quorum, 
otherwise it will hang waiting for that to happen.

Please make sure all your monitors are running.  If so, try running 
'ceph -s' on your cluster.  If that hangs as well, try accessing each 
monitor's admin socket to check what's happening [1].  If that too 
fails, try looking into the logs for something obviously wrong.  If you 
are not able to discern anything useful at that point, upload the logs 
to some place and point us to them -- we'll then be happy to take a look.

Hope this helps.

   -Joao

-- 
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com

Re: ceph-create-keys hung

[Abhay Sachan]

[-- Attachment #1.1: Type: text/plain, Size: 3312 bytes --]

Hi Joao,
Thanks for replying. All of my monitors are up and running and connected to
each other. "ceph -s" is failing on the cluster with following error:

2013-10-07 10:12:25.099261 7fd1b948d700 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2013-10-07 10:12:25.099271 7fd1b948d700  0 librados: client.admin
initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound

And the logs on each monitor has lots of entries like this:
NODE 1:

2013-10-07 03:58:51.153847 7ff2864c6700  0 mon.jul@0(probing).data_health(0)
update_stats avail 76% total 42332700 used 7901820 avail 32280480
2013-10-07 03:59:51.154051 7ff2864c6700  0 mon.jul@0(probing).data_health(0)
update_stats avail 76% total 42332700 used 7901832 avail 32280468
2013-10-07 04:00:51.154256 7ff2864c6700  0 mon.jul@0(probing).data_health(0)
update_stats avail 76% total 42332700 used 7901828 avail 32280472

NODE 2:
2013-10-07 10:12:10.345491 7fa6145b0700  0 mon.dec@0(probing).data_health(0)
update_stats avail 75% total 42332700 used 8199004 avail 31983296
2013-10-07 10:13:10.345677 7fa6145b0700  0 mon.dec@0(probing).data_health(0)
update_stats avail 75% total 42332700 used 8199004 avail 31983296
2013-10-07 10:14:10.345921 7fa6145b0700  0 mon.dec@0(probing).data_health(0)
update_stats avail 75% total 42332700 used 8199024 avail 31983276

NODE 3:

2013-10-07 10:13:00.880250 7fcd6459e700  0 mon.julilo@0(probing).data_health(0)
update_stats avail 35% total 42332700 used 25105920 avail
150763802013-10-07 10:14:00.880470 7fcd6459e700  0
mon.julilo@0(probing).data_health(0)
update_stats avail 35% total 42332700 used 25105924 avail 15076376
2013-10-07 10:15:00.880668 7fcd6459e700  0 mon.julilo@0(probing).data_health(0)
update_stats avail 35% total 42332700 used 25105924 avail 15076376

If you need some other logs, then please tell me how to enable/fetch them.
I will upload them someplace.


Regards,
Abhay

On Thu, Oct 3, 2013 at 8:31 PM, Joao Eduardo Luis <joao.luis-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>wrote:

> On 10/03/2013 02:44 PM, Abhay Sachan wrote:
>
>> Hi All,
>> I have tried setting up a ceph cluster with 3 nodes (3 monitors). I am
>> using RHEL 6.4 as OS with dumpling(0.67.3) release. Ceph cluster
>> creation (using ceph-deploy as well as mkcephfs), ceph-creates-keys
>> doesn't return on any of the servers. Whereas, if I create a cluster
>> with only 1 node (1 monitor), key creation goes through. Has anybody
>> seen this problem or any ideas what I might be missing??
>>
>> Regards,
>> Abhay
>>
>
> Those symptoms tell me that your monitors are not forming quorum.
> 'ceph-create-keys' needs the monitors to first establish a quorum,
> otherwise it will hang waiting for that to happen.
>
> Please make sure all your monitors are running.  If so, try running 'ceph
> -s' on your cluster.  If that hangs as well, try accessing each monitor's
> admin socket to check what's happening [1].  If that too fails, try looking
> into the logs for something obviously wrong.  If you are not able to
> discern anything useful at that point, upload the logs to some place and
> point us to them -- we'll then be happy to take a look.
>
> Hope this helps.
>
>   -Joao
>
> --
> Joao Eduardo Luis
> Software Engineer | http://inktank.com | http://ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 4377 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph-create-keys hung

[Joao Eduardo Luis]

On 08/10/13 05:40, Abhay Sachan wrote:
> Hi Joao,
> Thanks for replying. All of my monitors are up and running and connected
> to each other. "ceph -s" is failing on the cluster with following error:
>
> 2013-10-07 10:12:25.099261 7fd1b948d700 -1 monclient(hunting): ERROR:
> missing keyring, cannot use cephx for authentication
> 2013-10-07 10:12:25.099271 7fd1b948d700  0 librados: client.admin
> initialization error (2) No such file or directory
> Error connecting to cluster: ObjectNotFound

This says it all.  You're somehow missing your keyring.  Or maybe you're 
not but you're keeping it in a non-default location and you're not 
specifying a path to it either on your ceph.conf or via '--keyring <path>'.


> And the logs on each monitor has lots of entries like this:
> NODE 1:
>
> 2013-10-07 03:58:51.153847 7ff2864c6700  0
> mon.jul@0(probing).data_health(0) update_stats avail 76% total 42332700
> used 7901820 avail 32280480
> 2013-10-07 03:59:51.154051 7ff2864c6700  0
> mon.jul@0(probing).data_health(0) update_stats avail 76% total 42332700
> used 7901832 avail 32280468
> 2013-10-07 04:00:51.154256 7ff2864c6700  0
> mon.jul@0(probing).data_health(0) update_stats avail 76% total 42332700
> used 7901828 avail 32280472

Those messages are simply reports on the mon store's storage capacity.


   -Joao

>
>
> Regards,
> Abhay
>
> On Thu, Oct 3, 2013 at 8:31 PM, Joao Eduardo Luis <joao.luis-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org
> <mailto:joao.luis-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>> wrote:
>
>     On 10/03/2013 02:44 PM, Abhay Sachan wrote:
>
>         Hi All,
>         I have tried setting up a ceph cluster with 3 nodes (3
>         monitors). I am
>         using RHEL 6.4 as OS with dumpling(0.67.3) release. Ceph cluster
>         creation (using ceph-deploy as well as mkcephfs), ceph-creates-keys
>         doesn't return on any of the servers. Whereas, if I create a cluster
>         with only 1 node (1 monitor), key creation goes through. Has anybody
>         seen this problem or any ideas what I might be missing??
>
>         Regards,
>         Abhay
>
>
>     Those symptoms tell me that your monitors are not forming quorum.
>     'ceph-create-keys' needs the monitors to first establish a quorum,
>     otherwise it will hang waiting for that to happen.
>
>     Please make sure all your monitors are running.  If so, try running
>     'ceph -s' on your cluster.  If that hangs as well, try accessing
>     each monitor's admin socket to check what's happening [1].  If that
>     too fails, try looking into the logs for something obviously wrong.
>       If you are not able to discern anything useful at that point,
>     upload the logs to some place and point us to them -- we'll then be
>     happy to take a look.
>
>     Hope this helps.
>
>        -Joao
>
>     --
>     Joao Eduardo Luis
>     Software Engineer | http://inktank.com | http://ceph.com
>
>


-- 
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com

Re: ceph-create-keys hung

[Abhay Sachan]

[-- Attachment #1.1: Type: text/plain, Size: 7263 bytes --]

Hi Joao,
I gave the path for the keyring, and these messages are being printed on
the screen:

2013-10-07 15:18:55.048151 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400dfe0).fault
2013-10-07 15:18:58.048774 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4002290).fault
2013-10-07 15:19:01.049193 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b40038b0).fault
2013-10-07 15:19:04.049690 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.241:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4002590).fault
2013-10-07 15:19:07.050098 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b40038b0).fault
2013-10-07 15:19:10.050467 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4005710).fault
2013-10-07 15:19:13.051033 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.241:6789/0 pipe(0x7fa1b4000990 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4005870).fault
2013-10-07 15:19:16.051257 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b40042e0).fault
2013-10-07 15:19:19.051823 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.241:6789/0 pipe(0x7fa1b4000990 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4004990).fault
2013-10-07 15:19:22.052177 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b40042e0).fault
2013-10-07 15:19:25.052526 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.241:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400f4a0).fault
2013-10-07 15:19:28.053000 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b4007d90 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4007250).fault
2013-10-07 15:19:31.053452 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400f4a0).fault
2013-10-07 15:19:34.054070 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.241:6789/0 pipe(0x7fa1b4001cb0 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4007d90).fault
2013-10-07 15:19:37.054303 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400a7e0).fault
2013-10-07 15:19:40.054724 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.241:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400a1b0).fault
2013-10-07 15:19:43.055099 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4009610).fault
2013-10-07 15:19:46.055462 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4007d90).fault
2013-10-07 15:19:49.056004 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400c5a0).fault
2013-10-07 15:19:52.056492 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b4001cb0 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400b000).fault
2013-10-07 15:19:55.056740 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b4000990 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b40111b0).fault
2013-10-07 15:19:58.057117 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.241:6789/0 pipe(0x7fa1b4009110 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400b4a0).fault
2013-10-07 15:20:01.057596 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b400bc00 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400be60).fault
2013-10-07 15:20:04.058191 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b400d170 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b4015300).fault
2013-10-07 15:20:07.058837 7fa1c43bc700  0 -- :/1026989 >>
15.213.24.230:6789/0 pipe(0x7fa1b400f640 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b400bc00).fault
2013-10-07 15:20:10.059147 7fa1c42bb700  0 -- :/1026989 >>
15.213.24.231:6789/0 pipe(0x7fa1b400d170 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7fa1b40015c0).fault


Regards,
Abhay




On Tue, Oct 8, 2013 at 2:44 PM, Joao Eduardo Luis <joao.luis-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>wrote:

> On 08/10/13 05:40, Abhay Sachan wrote:
>
>> Hi Joao,
>> Thanks for replying. All of my monitors are up and running and connected
>> to each other. "ceph -s" is failing on the cluster with following error:
>>
>> 2013-10-07 10:12:25.099261 7fd1b948d700 -1 monclient(hunting): ERROR:
>> missing keyring, cannot use cephx for authentication
>> 2013-10-07 10:12:25.099271 7fd1b948d700  0 librados: client.admin
>> initialization error (2) No such file or directory
>> Error connecting to cluster: ObjectNotFound
>>
>
> This says it all.  You're somehow missing your keyring.  Or maybe you're
> not but you're keeping it in a non-default location and you're not
> specifying a path to it either on your ceph.conf or via '--keyring <path>'.
>
>
>
>  And the logs on each monitor has lots of entries like this:
>> NODE 1:
>>
>> 2013-10-07 03:58:51.153847 7ff2864c6700  0
>> mon.jul@0(probing).data_**health(0) update_stats avail 76% total 42332700
>> used 7901820 avail 32280480
>> 2013-10-07 03:59:51.154051 7ff2864c6700  0
>> mon.jul@0(probing).data_**health(0) update_stats avail 76% total 42332700
>> used 7901832 avail 32280468
>> 2013-10-07 04:00:51.154256 7ff2864c6700  0
>> mon.jul@0(probing).data_**health(0) update_stats avail 76% total 42332700
>> used 7901828 avail 32280472
>>
>
> Those messages are simply reports on the mon store's storage capacity.
>
>
>   -Joao
>
>
>>
>> Regards,
>> Abhay
>>
>> On Thu, Oct 3, 2013 at 8:31 PM, Joao Eduardo Luis <joao.luis-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org
>> <mailto:joao.luis-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>**> wrote:
>>
>>     On 10/03/2013 02:44 PM, Abhay Sachan wrote:
>>
>>         Hi All,
>>         I have tried setting up a ceph cluster with 3 nodes (3
>>         monitors). I am
>>         using RHEL 6.4 as OS with dumpling(0.67.3) release. Ceph cluster
>>         creation (using ceph-deploy as well as mkcephfs),
>> ceph-creates-keys
>>         doesn't return on any of the servers. Whereas, if I create a
>> cluster
>>         with only 1 node (1 monitor), key creation goes through. Has
>> anybody
>>         seen this problem or any ideas what I might be missing??
>>
>>         Regards,
>>         Abhay
>>
>>
>>     Those symptoms tell me that your monitors are not forming quorum.
>>     'ceph-create-keys' needs the monitors to first establish a quorum,
>>     otherwise it will hang waiting for that to happen.
>>
>>     Please make sure all your monitors are running.  If so, try running
>>     'ceph -s' on your cluster.  If that hangs as well, try accessing
>>     each monitor's admin socket to check what's happening [1].  If that
>>     too fails, try looking into the logs for something obviously wrong.
>>       If you are not able to discern anything useful at that point,
>>     upload the logs to some place and point us to them -- we'll then be
>>     happy to take a look.
>>
>>     Hope this helps.
>>
>>        -Joao
>>
>>     --
>>     Joao Eduardo Luis
>>     Software Engineer | http://inktank.com | http://ceph.com
>>
>>
>>
>
> --
> Joao Eduardo Luis
> Software Engineer | http://inktank.com | http://ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 10365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com