From: Chris Dunlop <chris@onthe.net.au>
To: Sage Weil <sage@inktank.com>
Cc: ceph-devel@vger.kernel.org
Subject: Re: Mon losing touch with OSDs
Date: Tue, 19 Feb 2013 14:02:03 +1100 [thread overview]
Message-ID: <20130219030202.GA5010@onthe.net.au> (raw)
In-Reply-To: <alpine.DEB.2.00.1302171743460.25660@cobra.newdream.net>
On Sun, Feb 17, 2013 at 05:44:29PM -0800, Sage Weil wrote:
> On Mon, 18 Feb 2013, Chris Dunlop wrote:
>> On Sat, Feb 16, 2013 at 09:05:21AM +1100, Chris Dunlop wrote:
>>> On Thu, Feb 14, 2013 at 08:57:11PM -0800, Sage Weil wrote:
>>>> On Fri, 15 Feb 2013, Chris Dunlop wrote:
>>>>> In an otherwise seemingly healthy cluster (ceph 0.56.2), what might cause the
>>>>> mons to lose touch with the osds?
>>>>
>>>> Can you enable 'debug ms = 1' on the mons and leave them that way, in the
>>>> hopes that this happens again? It will give us more information to go on.
>>>
>>> Debug turned on.
>>
>> We haven't experienced the cluster losing touch with the osds completely
>> since upgrading from 0.56.2 to 0.56.3, but we did lose touch with osd.1
>> for a few seconds before it recovered. See below for logs (reminder: 3
>> boxes, b2 is mon-only, b4 is mon+osd.0, b5 is mon+osd.1).
>
> Hrm, I don't see any obvious clues. You could enable 'debug ms = 1' on
> the osds as well. That will give us more to go on if/when it happens
> again, and should not affect performance significantly.
Done: ceph osd tell '*' injectargs '--debug-ms 1'
Now to wait for it to happen again.
Chris
next prev parent reply other threads:[~2013-02-19 3:02 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-02-15 3:29 Mon losing touch with OSDs Chris Dunlop
2013-02-15 4:57 ` Sage Weil
2013-02-15 22:05 ` Chris Dunlop
2013-02-17 23:41 ` Chris Dunlop
2013-02-18 1:44 ` Sage Weil
2013-02-19 3:02 ` Chris Dunlop [this message]
2013-02-20 2:07 ` Chris Dunlop
2013-02-22 3:06 ` Chris Dunlop
2013-02-22 21:57 ` Sage Weil
2013-02-22 23:35 ` Chris Dunlop
2013-02-22 23:43 ` Sage Weil
2013-02-23 0:08 ` Chris Dunlop
2013-02-23 0:13 ` Sage Weil
2013-02-23 0:25 ` Sage Weil
2013-02-23 0:50 ` Chris Dunlop
2013-02-23 1:10 ` Chris Dunlop
2013-02-23 0:57 ` Chris Dunlop
2013-02-23 1:30 ` Sage Weil
2013-02-23 1:49 ` Chris Dunlop
2013-02-23 1:52 ` Sage Weil
2013-02-23 2:02 ` Chris Dunlop
2013-03-01 2:02 ` Chris Dunlop
2013-03-01 5:00 ` Sage Weil
2013-03-08 3:12 ` Chris Dunlop
2013-03-08 22:47 ` Chris Dunlop
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130219030202.GA5010@onthe.net.au \
--to=chris@onthe.net.au \
--cc=ceph-devel@vger.kernel.org \
--cc=sage@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.