Re: Help...MDS Continuously Segfaulting

[Sam Lang <sam.lang@inktank.com>]

On 10/17/2012 11:23 AM, Nick Couchman wrote:
> Hmmm...I don't seem to have the dbg packages built...will have to go back and figure out how to build those.
>

Ah I thought you had installed from debian binaries.  If you compiled 
ceph yourself, to get the debugging symbols you have to reconfigure with 
-g on CXXFLAGS:

./configure CXXFLAGS=-g

-sam

> -Nick
>
>>>> On 2012/10/17 at 09:53, Sam Lang <sam.lang@inktank.com> wrote:
>> On 10/17/2012 09:42 AM, Nick Couchman wrote:
>>> Thanks...here's the backtrace:
>>> (gdb) bt
>>> #0  0x00000000004dcfea in ESession::replay(MDS*) ()
>>> #1  0x00000000006a2446 in MDLog::_replay_thread() ()
>>> #2  0x00000000004cf5ed in MDLog::ReplayThread::entry() ()
>>> #3  0x00007ffff764df05 in start_thread () from /lib64/libpthread.so.0
>>> #4  0x00007ffff680d10d in clone () from /lib64/libc.so.6
>>
>> Hi Nick,
>>
>> This doesn't have the debug symbols (line numbers in the source) we were
>> hoping for.  Could you install the ceph-dpg package and rerun?  You will
>> probably have to first uninstall the ceph package.
>>
>> Thanks,
>> -sam
>>
>>>
>>>>>> On 2012/10/17 at 07:34, Sam Lang <sam.lang@inktank.com> wrote:
>>>> On 10/16/2012 06:04 PM, Gregory Farnum wrote:
>>>>> Okay, that's the right debugging but it wasn't quite as helpful on its
>>>>> own as I expected. Can you get a core dump (you might already have
>>>>> one, depending on system settings) of the crash and open it up with
>>>>> gdb and get a full backtrace?
>>>>
>>>> You can also run the mds directly in gdb and avoid any core file ulimit
>>>> settings you have set:
>>>>
>>>>    > gdb --args ceph-mds -n mds.b -c /etc/ceph/ceph.conf -f
>>>> ...
>>>> (gdb) run
>>>>
>>>> Once you hit the segfault you can get the backtrace with:
>>>>
>>>> (gdb) bt
>>>>
>>>> -sam
>>>>
>>>>
>>>>> -Greg
>>>>>
>>>>> On Mon, Oct 15, 2012 at 10:59 AM, Nick Couchman <Nick.Couchman@seakr.com>
>>>> wrote:
>>>>>> Well, hopefully this is still okay...8.5MB bzip2d, 230MB unzipped.
>>>>>>
>>>>>> -Nick
>>>>>>
>>>>>>>>> On 2012/10/15 at 11:47, Gregory Farnum <greg@inktank.com> wrote:
>>>>>>> Yeah, zip it and post * somebody's going to have to download it and
>>>>>> do
>>>>>>> fun things. :)
>>>>>>> -Greg
>>>>>>>
>>>>>>> On Mon, Oct 15, 2012 at 10:43 AM, Nick Couchman
>>>>>> <Nick.Couchman@seakr.com>
>>>>>>> wrote:
>>>>>>>> Anywhere in particular I should make it available?  It's a little
>>>>>> over a
>>>>>>> million lines of debug in the file - I can put it on a pastebin, if
>>>>>> that
>>>>>>> works, or perhaps zip it up and throw it somewhere?
>>>>>>>>
>>>>>>>> -Nick
>>>>>>>>
>>>>>>>>>>> On 2012/10/15 at 11:26, Gregory Farnum <greg@inktank.com> wrote:
>>>>>>>>> Something in the MDS log is bad or is poking at a bug in the code.
>>>>>> Can
>>>>>>>>> you turn on MDS debugging and restart a daemon and put that log
>>>>>>>>> somewhere accessible?
>>>>>>>>> debug mds = 20
>>>>>>>>> debug journaler = 20
>>>>>>>>> debug ms = 1
>>>>>>>>> -Greg
>>>>>>>>>
>>>>>>>>> On Mon, Oct 15, 2012 at 10:02 AM, Nick Couchman
>>>>>> <Nick.Couchman@seakr.com>
>>>>>>>>> wrote:
>>>>>>>>>> Well, both of my MDSs seem to be down right now, and then
>>>>>> continually
>>>>>>>>> segfault (every time I try to start them) with the following:
>>>>>>>>>>
>>>>>>>>>> ceph-mdsmon-a:~ # ceph-mds -n mds.b -c /etc/ceph/ceph.conf -f
>>>>>>>>>> starting mds.b at :/0
>>>>>>>>>> *** Caught signal (Segmentation fault) **
>>>>>>>>>>     in thread 7fbe0d61d700
>>>>>>>>>>     ceph version 0.48.1argonaut
>>>>>>>>> (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
>>>>>>>>>>     1: ceph-mds() [0x7ef83a]
>>>>>>>>>>     2: (()+0xfd00) [0x7fbe15a0cd00]
>>>>>>>>>>     3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea]
>>>>>>>>>>     4: (MDLog::_replay_thread()+0x6b6) [0x6a2446]
>>>>>>>>>>     5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed]
>>>>>>>>>>     6: (()+0x7f05) [0x7fbe15a04f05]
>>>>>>>>>>     7: (clone()+0x6d) [0x7fbe14bc410d]
>>>>>>>>>> 2012-10-15 10:57:35.449161 7fbe0d61d700 -1 *** Caught signal
>>>>>> (Segmentation
>>>>>>>>> fault) **
>>>>>>>>>>     in thread 7fbe0d61d700
>>>>>>>>>>
>>>>>>>>>>     ceph version 0.48.1argonaut
>>>>>>>>> (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
>>>>>>>>>>     1: ceph-mds() [0x7ef83a]
>>>>>>>>>>     2: (()+0xfd00) [0x7fbe15a0cd00]
>>>>>>>>>>     3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea]
>>>>>>>>>>     4: (MDLog::_replay_thread()+0x6b6) [0x6a2446]
>>>>>>>>>>     5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed]
>>>>>>>>>>     6: (()+0x7f05) [0x7fbe15a04f05]
>>>>>>>>>>     7: (clone()+0x6d) [0x7fbe14bc410d]
>>>>>>>>>>     NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>>>>>> needed to
>>>>>>>>> interpret this.
>>>>>>>>>>
>>>>>>>>>>         0> 2012-10-15 10:57:35.449161 7fbe0d61d700 -1 *** Caught
>>>>>> signal
>>>>>>>>> (Segmentation fault) **
>>>>>>>>>>     in thread 7fbe0d61d700
>>>>>>>>>>
>>>>>>>>>>     ceph version 0.48.1argonaut
>>>>>>>>> (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
>>>>>>>>>>     1: ceph-mds() [0x7ef83a]
>>>>>>>>>>     2: (()+0xfd00) [0x7fbe15a0cd00]
>>>>>>>>>>     3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea]
>>>>>>>>>>     4: (MDLog::_replay_thread()+0x6b6) [0x6a2446]
>>>>>>>>>>     5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed]
>>>>>>>>>>     6: (()+0x7f05) [0x7fbe15a04f05]
>>>>>>>>>>     7: (clone()+0x6d) [0x7fbe14bc410d]
>>>>>>>>>>     NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>>>>>> needed to
>>>>>>>>> interpret this.
>>>>>>>>>>
>>>>>>>>>> Segmentation fault
>>>>>>>>>>
>>>>>>>>>> Anyone have any hints on recovering?  I'm running 0.48.1argonaut -
>>>>>> I can
>>>>>>>>> attempt to upgrade to 0.48.2 and see if that helps, but I figured
>>>>>> if anyone
>>>>>>>>> can offer any insight as to what to do to get the replay to run
>>>>>> without
>>>>>>>>> segfaulting?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --------
>>>>>>>>>> This e-mail may contain confidential and privileged material for
>>>>>> the sole use
>>>>>>>>> of the intended recipient.  If this email is not intended for you,
>>>>>> or you
>>>>>>> are
>>>>>>>>> not responsible for the delivery of this message to the intended
>>>>>> recipient,
>>>>>>>>> please note that this message may contain SEAKR Engineering
>>>>>> (SEAKR)
>>>>>>>>> Privileged/Proprietary Information.  In such a case, you are
>>>>>> strictly
>>>>>>>>> prohibited from downloading, photocopying, distributing or
>>>>>> otherwise using
>>>>>>>>> this message, its contents or attachments in any way.  If you have
>>>>>> received
>>>>>>>>> this message in error, please notify us immediately by replying to
>>>>>> this
>>>>>>> e-mail
>>>>>>>>> and delete the message from your mailbox.  Information contained in
>>>>>> this
>>>>>>>>> message that does not relate to the business of SEAKR is neither
>>>>>> endorsed by
>>>>>>>>> nor attributable to SEAKR.
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> ceph-devel" in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at
>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --------
>>>>>>>>
>>>>>>>> This e-mail may contain confidential and privileged material for the
>>>>>> sole use
>>>>>>> of the intended recipient.  If this email is not intended for you, or
>>>>>> you are
>>>>>>> not responsible for the delivery of this message to the intended
>>>>>> recipient,
>>>>>>> please note that this message may contain SEAKR Engineering (SEAKR)
>>>>>>> Privileged/Proprietary Information.  In such a case, you are strictly
>>>>>>
>>>>>>> prohibited from downloading, photocopying, distributing or otherwise
>>>>>> using
>>>>>>> this message, its contents or attachments in any way.  If you have
>>>>>> received
>>>>>>> this message in error, please notify us immediately by replying to
>>>>>> this e-mail
>>>>>>> and delete the message from your mailbox.  Information contained in
>>>>>> this
>>>>>>> message that does not relate to the business of SEAKR is neither
>>>>>> endorsed by
>>>>>>> nor attributable to SEAKR.
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>>>>> in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>>>
>>>>>> --------
>>>>>> This e-mail may contain confidential and privileged material for the sole use
>>>> of the intended recipient.  If this email is not intended for you, or you
>> are
>>>> not responsible for the delivery of this message to the intended recipient,
>>>> please note that this message may contain SEAKR Engineering (SEAKR)
>>>> Privileged/Proprietary Information.  In such a case, you are strictly
>>>> prohibited from downloading, photocopying, distributing or otherwise using
>>>> this message, its contents or attachments in any way.  If you have received
>>>> this message in error, please notify us immediately by replying to this
>> e-mail
>>>> and delete the message from your mailbox.  Information contained in this
>>>> message that does not relate to the business of SEAKR is neither endorsed by
>>>> nor attributable to SEAKR.
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>
>>>
>>>
>>> --------
>>> This e-mail may contain confidential and privileged material for the sole use
>> of the intended recipient.  If this email is not intended for you, or you are
>> not responsible for the delivery of this message to the intended recipient,
>> please note that this message may contain SEAKR Engineering (SEAKR)
>> Privileged/Proprietary Information.  In such a case, you are strictly
>> prohibited from downloading, photocopying, distributing or otherwise using
>> this message, its contents or attachments in any way.  If you have received
>> this message in error, please notify us immediately by replying to this e-mail
>> and delete the message from your mailbox.  Information contained in this
>> message that does not relate to the business of SEAKR is neither endorsed by
>> nor attributable to SEAKR.
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --------
> This e-mail may contain confidential and privileged material for the sole use of the intended recipient.  If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information.  In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way.  If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox.  Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR.
>