Cluster sync doesn't finsh

Cluster sync doesn't finsh

[Martin Mailand]

Hi,
I am doing cluster failure test, where I shut down one OSD an wait for 
the cluster to sync. But the sync never finshed, at around 4-5% it 
stops. I stoped osd2.

2011-11-17 16:40:48.015370    pg v1333: 600 pgs: 1 active, 546 
active+clean, 53 active+clean+degraded; 113 GB data, 183 GB used, 1142 
GB / 1395 GB avail; 4200/82404 degraded (5.097%)
2011-11-17 16:40:53.109391    pg v1334: 600 pgs: 1 active, 546 
active+clean, 53 active+clean+degraded; 113 GB data, 183 GB used, 1142 
GB / 1395 GB avail; 4117/82404 degraded (4.996%)
2011-11-17 16:40:58.228525    pg v1335: 600 pgs: 1 active, 546 
active+clean, 53 active+clean+degraded; 113 GB data, 183 GB used, 1142 
GB / 1395 GB avail; 4037/82404 degraded (4.899%)
2011-11-17 16:41:03.223778    pg v1336: 600 pgs: 547 active+clean, 53 
active+clean+degraded; 113 GB data, 183 GB used, 1142 GB / 1395 GB 
avail; 4025/82404 degraded (4.884%)
2011-11-17 16:42:45.520740    pg v1337: 600 pgs: 547 active+clean, 53 
active+clean+degraded; 113 GB data, 184 GB used, 1141 GB / 1395 GB 
avail; 4025/82404 degraded (4.884%)

^C
root@m-brick-000:~# date -R
Thu, 17 Nov 2011 17:56:08 +0100
root@m-brick-000:~#

So for the last hour nothing happend, there is no load on the cluster.

The osd log, the ceph.conf, pg dump, osd dump could be found here.

http://85.214.49.87/ceph/

ceph version 0.38-181-g2e19550 
(commit:2e195500b5d3a8ab8512bcf2a219a6b7ff922c97)

-martin

Re: Cluster sync doesn't finsh

[Tommi Virtanen]

On Thu, Nov 17, 2011 at 12:48, Martin Mailand <martin@tuxadero.com> wrote:
> Hi,
> I am doing cluster failure test, where I shut down one OSD an wait for the
> cluster to sync. But the sync never finshed, at around 4-5% it stops. I
> stoped osd2.
...
> 2011-11-17 16:42:45.520740    pg v1337: 600 pgs: 547 active+clean, 53
> active+clean+degraded; 113 GB data, 184 GB used, 1141 GB / 1395 GB avail;
> 4025/82404 degraded (4.884%)
...
> The osd log, the ceph.conf, pg dump, osd dump could be found here.
>
> http://85.214.49.87/ceph/

This looks a bit worrying:

2011-11-17 17:56:35.771574 7f704c834700 -- 192.168.42.113:0/2424 >>
192.168.42.114:6802/21115 pipe(0x2596c80 sd=17 pgs=0 cs=0 l=0).connect
claims to be 192.168.42.114:6802/21507 not 192.168.42.114:6802/21115 -
wrong node!

So osd.0 is basically refusing to talk to one of the other OSDs. I
don't understand the messenger well enough to know why this would be,
but it wouldn't surprise me if this problem kept the objects degraded
-- it looks like a breakage in the osd<->osd communication.

Now if this was the reason, I'd expect a restart of all the OSDs to
get it back in shape; messenger state is ephemeral. Can you confirm
that?
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Cluster sync doesn't finsh

[Gregory Farnum]

On Fri, Nov 18, 2011 at 10:05 AM, Tommi Virtanen
<tommi.virtanen@dreamhost.com> wrote:
> On Thu, Nov 17, 2011 at 12:48, Martin Mailand <martin@tuxadero.com> wrote:
>> Hi,
>> I am doing cluster failure test, where I shut down one OSD an wait for the
>> cluster to sync. But the sync never finshed, at around 4-5% it stops. I
>> stoped osd2.
> ...
>> 2011-11-17 16:42:45.520740    pg v1337: 600 pgs: 547 active+clean, 53
>> active+clean+degraded; 113 GB data, 184 GB used, 1141 GB / 1395 GB avail;
>> 4025/82404 degraded (4.884%)
> ...
>> The osd log, the ceph.conf, pg dump, osd dump could be found here.
>>
>> http://85.214.49.87/ceph/
>
> This looks a bit worrying:
>
> 2011-11-17 17:56:35.771574 7f704c834700 -- 192.168.42.113:0/2424 >>
> 192.168.42.114:6802/21115 pipe(0x2596c80 sd=17 pgs=0 cs=0 l=0).connect
> claims to be 192.168.42.114:6802/21507 not 192.168.42.114:6802/21115 -
> wrong node!
>
> So osd.0 is basically refusing to talk to one of the other OSDs. I
> don't understand the messenger well enough to know why this would be,
> but it wouldn't surprise me if this problem kept the objects degraded
> -- it looks like a breakage in the osd<->osd communication.
>
> Now if this was the reason, I'd expect a restart of all the OSDs to
> get it back in shape; messenger state is ephemeral. Can you confirm
> that?

Probably not — that wrong node thing can occur for a lot of different
reasons, some of which matter and most of which don't. Sam's looking
into the problem; there's something going wrong with the CRUSH
calculations or the monitor PG placement overrides or something...
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Cluster sync doesn't finsh

[Samuel Just]

It looks like a crushmap related problem.  Could you send us the crushmap?

ceph osd getcrushmap

Thanks
-Sam

On Fri, Nov 18, 2011 at 10:13 AM, Gregory Farnum
<gregory.farnum@dreamhost.com> wrote:
>
> On Fri, Nov 18, 2011 at 10:05 AM, Tommi Virtanen
> <tommi.virtanen@dreamhost.com> wrote:
> > On Thu, Nov 17, 2011 at 12:48, Martin Mailand <martin@tuxadero.com> wrote:
> >> Hi,
> >> I am doing cluster failure test, where I shut down one OSD an wait for the
> >> cluster to sync. But the sync never finshed, at around 4-5% it stops. I
> >> stoped osd2.
> > ...
> >> 2011-11-17 16:42:45.520740    pg v1337: 600 pgs: 547 active+clean, 53
> >> active+clean+degraded; 113 GB data, 184 GB used, 1141 GB / 1395 GB avail;
> >> 4025/82404 degraded (4.884%)
> > ...
> >> The osd log, the ceph.conf, pg dump, osd dump could be found here.
> >>
> >> http://85.214.49.87/ceph/
> >
> > This looks a bit worrying:
> >
> > 2011-11-17 17:56:35.771574 7f704c834700 -- 192.168.42.113:0/2424 >>
> > 192.168.42.114:6802/21115 pipe(0x2596c80 sd=17 pgs=0 cs=0 l=0).connect
> > claims to be 192.168.42.114:6802/21507 not 192.168.42.114:6802/21115 -
> > wrong node!
> >
> > So osd.0 is basically refusing to talk to one of the other OSDs. I
> > don't understand the messenger well enough to know why this would be,
> > but it wouldn't surprise me if this problem kept the objects degraded
> > -- it looks like a breakage in the osd<->osd communication.
> >
> > Now if this was the reason, I'd expect a restart of all the OSDs to
> > get it back in shape; messenger state is ephemeral. Can you confirm
> > that?
>
> Probably not — that wrong node thing can occur for a lot of different
> reasons, some of which matter and most of which don't. Sam's looking
> into the problem; there's something going wrong with the CRUSH
> calculations or the monitor PG placement overrides or something...
> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Cluster sync doesn't finsh

[Martin Mailand]

Hi Sam,

here the crushmap

http://85.214.49.87/ceph/crushmap.txt
http://85.214.49.87/ceph/crushmap

-martin

Samuel Just schrieb:
> It looks like a crushmap related problem.  Could you send us the crushmap?
> 
> ceph osd getcrushmap
> 
> Thanks
> -Sam
> 
> On Fri, Nov 18, 2011 at 10:13 AM, Gregory Farnum
> <gregory.farnum@dreamhost.com> wrote:
>> On Fri, Nov 18, 2011 at 10:05 AM, Tommi Virtanen
>> <tommi.virtanen@dreamhost.com> wrote:
>>> On Thu, Nov 17, 2011 at 12:48, Martin Mailand <martin@tuxadero.com> wrote:
>>>> Hi,
>>>> I am doing cluster failure test, where I shut down one OSD an wait for the
>>>> cluster to sync. But the sync never finshed, at around 4-5% it stops. I
>>>> stoped osd2.
>>> ...
>>>> 2011-11-17 16:42:45.520740    pg v1337: 600 pgs: 547 active+clean, 53
>>>> active+clean+degraded; 113 GB data, 184 GB used, 1141 GB / 1395 GB avail;
>>>> 4025/82404 degraded (4.884%)
>>> ...
>>>> The osd log, the ceph.conf, pg dump, osd dump could be found here.
>>>>
>>>> http://85.214.49.87/ceph/
>>> This looks a bit worrying:
>>>
>>> 2011-11-17 17:56:35.771574 7f704c834700 -- 192.168.42.113:0/2424 >>
>>> 192.168.42.114:6802/21115 pipe(0x2596c80 sd=17 pgs=0 cs=0 l=0).connect
>>> claims to be 192.168.42.114:6802/21507 not 192.168.42.114:6802/21115 -
>>> wrong node!
>>>
>>> So osd.0 is basically refusing to talk to one of the other OSDs. I
>>> don't understand the messenger well enough to know why this would be,
>>> but it wouldn't surprise me if this problem kept the objects degraded
>>> -- it looks like a breakage in the osd<->osd communication.
>>>
>>> Now if this was the reason, I'd expect a restart of all the OSDs to
>>> get it back in shape; messenger state is ephemeral. Can you confirm
>>> that?
>> Probably not — that wrong node thing can occur for a lot of different
>> reasons, some of which matter and most of which don't. Sam's looking
>> into the problem; there's something going wrong with the CRUSH
>> calculations or the monitor PG placement overrides or something...
>> -Greg
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Cluster sync doesn't finsh

[Samuel Just]

I've inserted this bug as #1738.  Unfortunately, this will take a bit
of effort to fix.  In the short term, you could switch to a crushmap
where each node at the bottom level of the hierarchy contains more
than one device.  (i.e., remove the node level and stop at the rack
level).

Thanks for the help!
-Sam

On Fri, Nov 18, 2011 at 12:17 PM, Martin Mailand <martin@tuxadero.com> wrote:
> Hi Sam,
>
> here the crushmap
>
> http://85.214.49.87/ceph/crushmap.txt
> http://85.214.49.87/ceph/crushmap
>
> -martin
>
> Samuel Just schrieb:
>>
>> It looks like a crushmap related problem.  Could you send us the crushmap?
>>
>> ceph osd getcrushmap
>>
>> Thanks
>> -Sam
>>
>> On Fri, Nov 18, 2011 at 10:13 AM, Gregory Farnum
>> <gregory.farnum@dreamhost.com> wrote:
>>>
>>> On Fri, Nov 18, 2011 at 10:05 AM, Tommi Virtanen
>>> <tommi.virtanen@dreamhost.com> wrote:
>>>>
>>>> On Thu, Nov 17, 2011 at 12:48, Martin Mailand <martin@tuxadero.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>> I am doing cluster failure test, where I shut down one OSD an wait for
>>>>> the
>>>>> cluster to sync. But the sync never finshed, at around 4-5% it stops. I
>>>>> stoped osd2.
>>>>
>>>> ...
>>>>>
>>>>> 2011-11-17 16:42:45.520740    pg v1337: 600 pgs: 547 active+clean, 53
>>>>> active+clean+degraded; 113 GB data, 184 GB used, 1141 GB / 1395 GB
>>>>> avail;
>>>>> 4025/82404 degraded (4.884%)
>>>>
>>>> ...
>>>>>
>>>>> The osd log, the ceph.conf, pg dump, osd dump could be found here.
>>>>>
>>>>> http://85.214.49.87/ceph/
>>>>
>>>> This looks a bit worrying:
>>>>
>>>> 2011-11-17 17:56:35.771574 7f704c834700 -- 192.168.42.113:0/2424 >>
>>>> 192.168.42.114:6802/21115 pipe(0x2596c80 sd=17 pgs=0 cs=0 l=0).connect
>>>> claims to be 192.168.42.114:6802/21507 not 192.168.42.114:6802/21115 -
>>>> wrong node!
>>>>
>>>> So osd.0 is basically refusing to talk to one of the other OSDs. I
>>>> don't understand the messenger well enough to know why this would be,
>>>> but it wouldn't surprise me if this problem kept the objects degraded
>>>> -- it looks like a breakage in the osd<->osd communication.
>>>>
>>>> Now if this was the reason, I'd expect a restart of all the OSDs to
>>>> get it back in shape; messenger state is ephemeral. Can you confirm
>>>> that?
>>>
>>> Probably not — that wrong node thing can occur for a lot of different
>>> reasons, some of which matter and most of which don't. Sam's looking
>>> into the problem; there's something going wrong with the CRUSH
>>> calculations or the monitor PG placement overrides or something...
>>> -Greg
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Cluster sync doesn't finsh

[Martin Mailand]

Hi Sam,
is there anything new on this Issue, which I could test?

-martin


Am 19.11.2011 02:05, schrieb Samuel Just:
> I've inserted this bug as #1738.  Unfortunately, this will take a bit
> of effort to fix.  In the short term, you could switch to a crushmap
> where each node at the bottom level of the hierarchy contains more
> than one device.  (i.e., remove the node level and stop at the rack
> level).
>
> Thanks for the help!
> -Sam
>
> On Fri, Nov 18, 2011 at 12:17 PM, Martin Mailand<martin@tuxadero.com>  wrote:
>> Hi Sam,
>>
>> here the crushmap
>>
>> http://85.214.49.87/ceph/crushmap.txt
>> http://85.214.49.87/ceph/crushmap
>>
>> -martin
>>
>> Samuel Just schrieb:
>>>
>>> It looks like a crushmap related problem.  Could you send us the crushmap?
>>>
>>> ceph osd getcrushmap
>>>
>>> Thanks
>>> -Sam
>>>
>>> On Fri, Nov 18, 2011 at 10:13 AM, Gregory Farnum
>>> <gregory.farnum@dreamhost.com>  wrote:
>>>>
>>>> On Fri, Nov 18, 2011 at 10:05 AM, Tommi Virtanen
>>>> <tommi.virtanen@dreamhost.com>  wrote:
>>>>>
>>>>> On Thu, Nov 17, 2011 at 12:48, Martin Mailand<martin@tuxadero.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I am doing cluster failure test, where I shut down one OSD an wait for
>>>>>> the
>>>>>> cluster to sync. But the sync never finshed, at around 4-5% it stops. I
>>>>>> stoped osd2.
>>>>>
>>>>> ...
>>>>>>
>>>>>> 2011-11-17 16:42:45.520740    pg v1337: 600 pgs: 547 active+clean, 53
>>>>>> active+clean+degraded; 113 GB data, 184 GB used, 1141 GB / 1395 GB
>>>>>> avail;
>>>>>> 4025/82404 degraded (4.884%)
>>>>>
>>>>> ...
>>>>>>
>>>>>> The osd log, the ceph.conf, pg dump, osd dump could be found here.
>>>>>>
>>>>>> http://85.214.49.87/ceph/
>>>>>
>>>>> This looks a bit worrying:
>>>>>
>>>>> 2011-11-17 17:56:35.771574 7f704c834700 -- 192.168.42.113:0/2424>>
>>>>> 192.168.42.114:6802/21115 pipe(0x2596c80 sd=17 pgs=0 cs=0 l=0).connect
>>>>> claims to be 192.168.42.114:6802/21507 not 192.168.42.114:6802/21115 -
>>>>> wrong node!
>>>>>
>>>>> So osd.0 is basically refusing to talk to one of the other OSDs. I
>>>>> don't understand the messenger well enough to know why this would be,
>>>>> but it wouldn't surprise me if this problem kept the objects degraded
>>>>> -- it looks like a breakage in the osd<->osd communication.
>>>>>
>>>>> Now if this was the reason, I'd expect a restart of all the OSDs to
>>>>> get it back in shape; messenger state is ephemeral. Can you confirm
>>>>> that?
>>>>
>>>> Probably not — that wrong node thing can occur for a lot of different
>>>> reasons, some of which matter and most of which don't. Sam's looking
>>>> into the problem; there's something going wrong with the CRUSH
>>>> calculations or the monitor PG placement overrides or something...
>>>> -Greg
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html