Re: mon crash on debian wheezy

[Xiaopong Tran <xiaopong.tran@gmail.com>]

On 08/29/2012 12:21 AM, Gregory Farnum wrote:
> On Tue, Aug 28, 2012 at 7:50 AM, Xiaopong Tran <xiaopong.tran@gmail.com> wrote:
>> On 08/25/2012 12:28 AM, Sage Weil wrote:
>>>
>>> On Fri, 24 Aug 2012, Xiaopong Tran wrote:
>>>>
>>>> Hello,
>>>>
>>>> I've been running the 0.48argonaut on production for over a month
>>>> without any issue. and today, I suddenly lost one mon. Taking a look
>>>> into the syslog file, I see the following trace log. I just couldn't
>>>> see what's wrong from the trace log. However, this event created
>>>> a gigantic core file. Here's the size of the core file:
>>>>
>>>> -rw------- 1 root root 16085647360 Aug 24 14:53 core
>>>>
>>>> This happened while we were migrating data from our old storage
>>>> to the ceph. We are running about 20 processes, migrating data
>>>> into ceph, while there are about 30 more application processes
>>>> reading from and writing new data to it.
>>>>
>>>> The following is from syslog:
>>>
>>>
>>> We've seen these backtraces before too, but haven't figured out what
>>> causes them.  (See, for example, http://tracker.newdream.net/issues/2026.)
>>>
>>> Was there anything in the mon's log file?  In most cases, a crash results
>>> in a stack trace of ceph-mon in the mon log file.
>>>
>>> Glad to hear everything recovered nicely afterwards.  :)
>>>
>>> Thanks!
>>> sage
>>>
>>
>> Ah well, I got two crashes in less than 3 days. I browsed thru the
>> mon log files, and the ceph log files, and there is nothing suspicious,
>> no trace dump or anything.
>>
>> One question I don't get is, after mon has crashed, it's not running
>> anymore, who is creating that empty mon log? The same question goes
>> for osd. I had two osd down today, and I also see empty osd log files.
>>
>> And how does the crash end up generating such a huge core file?
>>
>> If there's any information I can provide, I'd be happy to do so.
>
> Can you extract the backtrace from the core dump?
>

Will try to do that, it's a big one though :)

Thanks

Xiaopong